[jira] [Created] (HBASE-27425) Run flaky test job more often
Sean Busbey created HBASE-27425: --- Summary: Run flaky test job more often Key: HBASE-27425 URL: https://issues.apache.org/jira/browse/HBASE-27425 Project: HBase Issue Type: Task Components: test Reporter: Sean Busbey Attachments: ci-hbase-hbase-flaky-test-master-time-png.png, ci-hbase-hbase-load-chart-long.png We currently run the flaky tests once per 4 hours and keep a backlog of 50 runs. That means it takes a perfect test a bit over 8 days to get back out of being penalized. we regularly are leaving a bunch of executors idle (see attached PNG) and I think we should run the tests more often. we use an executor per branch for a flaky test run. the duration depends on the specific tests that are flagged, but it has a timeout of 2 hours. as an example the master branch tests currently take < 30 minutes (see other attached PNG). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-24457) release scripts on mac os are too slow
[ https://issues.apache.org/jira/browse/HBASE-24457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-24457. - Fix Version/s: 3.0.0-alpha-1 (was: 3.0.0-alpha-4) Resolution: Fixed > release scripts on mac os are too slow > -- > > Key: HBASE-24457 > URL: https://issues.apache.org/jira/browse/HBASE-24457 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > related changes to try to bring the time down to something I can use -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-26983) Upgrade JRuby to 9.3.4.0
[
https://issues.apache.org/jira/browse/HBASE-26983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-26983.
-
Resolution: Fixed
> Upgrade JRuby to 9.3.4.0
>
>
> Key: HBASE-26983
> URL: https://issues.apache.org/jira/browse/HBASE-26983
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Affects Versions: 3.0.0-alpha-2, 2.4.11
> Environment: Apple M1 OSX ARM64.
>Reporter: Vijay Akkineni
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> Hbase shell is failing to start on Apple M1 OSX ARM 64 processor architecture.
> *Error:*
> {code}
> Version 2.4.11, r7e672a0da0586e6b7449310815182695bc6ae193, Tue Mar 15
> 10:31:00 PDT 2022
> Took 0.0010 seconds
> NotImplementedError: fstat unimplemented unsupported or native support failed
> to load; see https://github.com/jruby/jruby/wiki/Native-Libraries
> initialize at org/jruby/RubyIO.java:1015
> open at org/jruby/RubyIO.java:1156
> initialize at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/input-method.rb:141
> initialize at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/context.rb:70
> initialize at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:410
> initialize at uri:classloader:/irb/hirb.rb:49
> at classpath:/jar-bootstrap.rb:223
> {code}
>
> {*}Uname output{*}:
> {code}
> Darwin vijays-mbp.lan 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18
> 00:46:32 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T6000 arm64
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (HBASE-27106) HBase site generation should use its own JRuby
Sean Busbey created HBASE-27106:
---
Summary: HBase site generation should use its own JRuby
Key: HBASE-27106
URL: https://issues.apache.org/jira/browse/HBASE-27106
Project: HBase
Issue Type: Bug
Components: website
Reporter: Sean Busbey
Assignee: Sean Busbey
The maven plugin we use to generate the reference guide,
{{asciidoctor-maven-plugin}}, does not yet work with JRuby 9.3.
Due to a dependency error we currently get a mix of JRuby 9.2 and 9.3 libraries
when building the site goal, which fails without an actionable error message.
Correcting the dependency to consistently get JRuby 9.3 also results in
failure, but due to gem resolution.
We originally started overriding the JRuby version to get Aarch64 support. The
version that ships with the plugin has this support already.
Falling back to JRuby 9.2 for the plugin will mean our reference guide building
won't work on M1 systems, but I don't see how we can change that until the
plugin updates for JRuby 9.3 changes.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Reopened] (HBASE-26983) Upgrade JRuby to 9.3.4.0
[
https://issues.apache.org/jira/browse/HBASE-26983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reopened HBASE-26983:
-
reopened to evaluate impact on site build
> Upgrade JRuby to 9.3.4.0
>
>
> Key: HBASE-26983
> URL: https://issues.apache.org/jira/browse/HBASE-26983
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Affects Versions: 3.0.0-alpha-2, 2.4.11
> Environment: Apple M1 OSX ARM64.
>Reporter: Vijay Akkineni
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> Hbase shell is failing to start on Apple M1 OSX ARM 64 processor architecture.
> *Error:*
> {code}
> Version 2.4.11, r7e672a0da0586e6b7449310815182695bc6ae193, Tue Mar 15
> 10:31:00 PDT 2022
> Took 0.0010 seconds
> NotImplementedError: fstat unimplemented unsupported or native support failed
> to load; see https://github.com/jruby/jruby/wiki/Native-Libraries
> initialize at org/jruby/RubyIO.java:1015
> open at org/jruby/RubyIO.java:1156
> initialize at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/input-method.rb:141
> initialize at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/context.rb:70
> initialize at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:410
> initialize at uri:classloader:/irb/hirb.rb:49
> at classpath:/jar-bootstrap.rb:223
> {code}
>
> {*}Uname output{*}:
> {code}
> Darwin vijays-mbp.lan 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18
> 00:46:32 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T6000 arm64
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-26983) Upgrade JRuby to 9.3.4.0
[
https://issues.apache.org/jira/browse/HBASE-26983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-26983.
-
Release Note:
Updates the version of JRuby that ships with HBase for the HBase shell to
9.3.4.0.
Note that this changes the supported version of Ruby for HBase shell
integration to Ruby 2.6.
Resolution: Fixed
> Upgrade JRuby to 9.3.4.0
>
>
> Key: HBASE-26983
> URL: https://issues.apache.org/jira/browse/HBASE-26983
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Affects Versions: 3.0.0-alpha-2, 2.4.11
> Environment: Apple M1 OSX ARM64.
>Reporter: Vijay Akkineni
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> Hbase shell is failing to start on Apple M1 OSX ARM 64 processor architecture.
> *Error:*
> {code}
> Version 2.4.11, r7e672a0da0586e6b7449310815182695bc6ae193, Tue Mar 15
> 10:31:00 PDT 2022
> Took 0.0010 seconds
> NotImplementedError: fstat unimplemented unsupported or native support failed
> to load; see https://github.com/jruby/jruby/wiki/Native-Libraries
> initialize at org/jruby/RubyIO.java:1015
> open at org/jruby/RubyIO.java:1156
> initialize at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/input-method.rb:141
> initialize at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/context.rb:70
> initialize at
> uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:410
> initialize at uri:classloader:/irb/hirb.rb:49
> at classpath:/jar-bootstrap.rb:223
> {code}
>
> {*}Uname output{*}:
> {code}
> Darwin vijays-mbp.lan 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18
> 00:46:32 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T6000 arm64
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-27006) cordon off large ci worker nodes
[ https://issues.apache.org/jira/browse/HBASE-27006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-27006. - Fix Version/s: 2.5.0 2.6.0 3.0.0-alpha-3 2.4.13 Assignee: Sean Busbey Resolution: Fixed > cordon off large ci worker nodes > > > Key: HBASE-27006 > URL: https://issues.apache.org/jira/browse/HBASE-27006 > Project: HBase > Issue Type: Sub-task >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.13 > > > Make sure our node labeling keeps test workloads that off of the new large > spec nodes so that they're only used for the nightly k8s based tests -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27012) nightly ITBLL
Sean Busbey created HBASE-27012: --- Summary: nightly ITBLL Key: HBASE-27012 URL: https://issues.apache.org/jira/browse/HBASE-27012 Project: HBase Issue Type: Sub-task Reporter: Sean Busbey -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27010) container images for HBase
Sean Busbey created HBASE-27010: --- Summary: container images for HBase Key: HBASE-27010 URL: https://issues.apache.org/jira/browse/HBASE-27010 Project: HBase Issue Type: Sub-task Reporter: Sean Busbey we need container images of the build that got made during the nightly run -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27011) migrate existing nightly test
Sean Busbey created HBASE-27011: --- Summary: migrate existing nightly test Key: HBASE-27011 URL: https://issues.apache.org/jira/browse/HBASE-27011 Project: HBase Issue Type: Sub-task Reporter: Sean Busbey once we can run a cluster on k8s in the nightly worker node, migrate our existing test to use it. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27009) container images for ZooKeeper
Sean Busbey created HBASE-27009: --- Summary: container images for ZooKeeper Key: HBASE-27009 URL: https://issues.apache.org/jira/browse/HBASE-27009 Project: HBase Issue Type: Sub-task Reporter: Sean Busbey we need image(s) for a zk cluster for our nightly cluster run -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27008) container images for Hadoop
Sean Busbey created HBASE-27008: --- Summary: container images for Hadoop Key: HBASE-27008 URL: https://issues.apache.org/jira/browse/HBASE-27008 Project: HBase Issue Type: Sub-task Reporter: Sean Busbey we need image(s) for the hadoop cluster we run the nightly tests on top of -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27007) nightly support scripts for install/setup of single-node k8s env
Sean Busbey created HBASE-27007: --- Summary: nightly support scripts for install/setup of single-node k8s env Key: HBASE-27007 URL: https://issues.apache.org/jira/browse/HBASE-27007 Project: HBase Issue Type: Sub-task Reporter: Sean Busbey minikube , kind, something like that. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27006) cordon off large ci worker nodes
Sean Busbey created HBASE-27006: --- Summary: cordon off large ci worker nodes Key: HBASE-27006 URL: https://issues.apache.org/jira/browse/HBASE-27006 Project: HBase Issue Type: Sub-task Reporter: Sean Busbey Make sure our node labeling keeps test workloads that off of the new large spec nodes so that they're only used for the nightly k8s based tests -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27005) move nightly cluster tests to run on k8s
Sean Busbey created HBASE-27005: --- Summary: move nightly cluster tests to run on k8s Key: HBASE-27005 URL: https://issues.apache.org/jira/browse/HBASE-27005 Project: HBase Issue Type: Umbrella Components: integration tests Reporter: Sean Busbey -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-26936) project Jenkins node needs a fail-safe clean up of surefire deferred output
Sean Busbey created HBASE-26936: --- Summary: project Jenkins node needs a fail-safe clean up of surefire deferred output Key: HBASE-26936 URL: https://issues.apache.org/jira/browse/HBASE-26936 Project: HBase Issue Type: Task Components: test Reporter: Sean Busbey Assignee: Sean Busbey We've had build nodes run out of space a few times now caused by surefire deferred stdout / stderr filling a disk. Come up with a strategy to make recovery from this automated. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26889) nightly yetus tests don't reflect failure when required environment variables are missing.
Sean Busbey created HBASE-26889:
---
Summary: nightly yetus tests don't reflect failure when required
environment variables are missing.
Key: HBASE-26889
URL: https://issues.apache.org/jira/browse/HBASE-26889
Project: HBase
Issue Type: Bug
Components: test
Reporter: Sean Busbey
the nightly yetus wrapper shows an error when we're missing some needed
environment variables, but the stage shows success still.
e.g
{code:java}
[2022-03-24T22:35:39.450Z] [ERROR] Required environment variable 'DEBUG' is not
set.
[2022-03-24T22:35:39.451Z] [ERROR] Required environment variable
'USE_YETUS_PRERELEASE' is not set.
[2022-03-24T22:35:39.451Z] [ERROR] Please set the required environment
variables before invoking. If this error is on Jenkins, then please file a
JIRA about the error. {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26888) Update branch-specific nightly test handling to allow for earlier release line feature branches
Sean Busbey created HBASE-26888:
---
Summary: Update branch-specific nightly test handling to allow for
earlier release line feature branches
Key: HBASE-26888
URL: https://issues.apache.org/jira/browse/HBASE-26888
Project: HBase
Issue Type: Task
Components: integration tests, test
Reporter: Sean Busbey
due to limitations in ASF infra our branch protection rules prohibit force
pushes or deletions on branches that start with "master" or "branch-".
Our current nightly tests assume that feature branches specific to branch-2 or
branch-1 will start with the release branch name rather than the feature Jira
key. However, doing so would mean we need to file INFRA jiras to clean up
afterwards.
We can't change the limitations of branch protection rules, so we should update
the nightly tests to allow for feature branches named like
{{{}HBASE-X-branch-2{}}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26887) nightly integration test claims success while showing error parsing shell output
Sean Busbey created HBASE-26887:
---
Summary: nightly integration test claims success while showing
error parsing shell output
Key: HBASE-26887
URL: https://issues.apache.org/jira/browse/HBASE-26887
Project: HBase
Issue Type: Bug
Components: integration tests, shell
Reporter: Sean Busbey
current master branch claims that the integration test is succeeding, but
looking at the detailed output shows a failure to parse the shell output:
{code:java}
Starting up Hadoop
waiting for Hadoop to finish starting up.
waiting for Hadoop to finish starting up.
waiting for Hadoop to finish starting up.
waiting for Hadoop to finish starting up.
Verifying configs
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/output-integration/hadoop-3/hbase-conf/core-site.xml:
valid
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/output-integration/hadoop-3/hbase-conf/hbase-site.xml:
valid
OK
Listing HDFS contents
Starting up HBase
running master, logging to
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/hbase-install/bin/../logs/hbase-jenkins-master-jenkins-hbase12.out
retry waiting for hbase to come up.
Setting up table 'test:example' with 1,000 regions
writing out example TSV to example.tsv
uploading example.tsv to HDFS
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
Importing TSV via shaded client artifact for HBase - MapReduce integration.
Verifying row count from import.
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/component/dev-support/hbase_nightly_pseudo-distributed-test.sh:
line 418: [: hbase:002:0> : integer expression expected
Hadoop client jars not given; getting them from 'hadoop classpath' for the
example.
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
Building shaded client example.
Running shaded client example. It'll fetch the set of regions, round-trip them
to a file in HDFS, then write them one-per-row into the test table.
Checking on results of example program.
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
Verifying row count from example.
/home/jenkins/jenkins-home/workspace/HBase_Nightly_master/component/dev-support/hbase_nightly_pseudo-distributed-test.sh:
line 531: [: hbase:002:0> : integer expression expected
ERROR: Only found hbase:002:0> rows.
Shutting down HBase {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26704) EOL HBase 2.3.x
[ https://issues.apache.org/jira/browse/HBASE-26704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-26704. - Resolution: Fixed > EOL HBase 2.3.x > --- > > Key: HBASE-26704 > URL: https://issues.apache.org/jira/browse/HBASE-26704 > Project: HBase > Issue Type: Task > Components: community >Reporter: Sean Busbey >Priority: Major > > VOTE thread is here: > [https://lists.apache.org/thread/v3gfoyf255rpvql4h3g68ong5p4ps4rs] > * remove from [https://hbase.apache.org/downloads] > * delete from dist svn repo > [https://dist.apache.org/repos/dist/release/hbase/] (which will remove the > release from downloads.apache.org) > * send an announcement to user@hbase and dev@hbase > * delete Jenkins jobs -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26823) ci-hbase expansion
Sean Busbey created HBASE-26823: --- Summary: ci-hbase expansion Key: HBASE-26823 URL: https://issues.apache.org/jira/browse/HBASE-26823 Project: HBase Issue Type: Task Components: community, integration tests, test Reporter: Sean Busbey Assignee: Sean Busbey I need a place to host images related to a dev@hbase discussion about how ci-hbase is going an what we as a project would like to expand into. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26546) hbase-shaded-client missing required thirdparty classes under hadoop 3.3.1
[
https://issues.apache.org/jira/browse/HBASE-26546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-26546.
-
Resolution: Fixed
> hbase-shaded-client missing required thirdparty classes under hadoop 3.3.1
> --
>
> Key: HBASE-26546
> URL: https://issues.apache.org/jira/browse/HBASE-26546
> Project: HBase
> Issue Type: Bug
> Components: Client, hadoop3, shading
>Affects Versions: 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8, 2.4.9
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.10
>
>
> In HBASE-25792, the shaded thirdparty libraries from hadoop were removed from
> the hbase-shaded-client fat jar to satisfy invariant checks. Unfortunately
> this causes users of hbase-shaded-client to fail, because required classes
> are not available at runtime.
> The specific failure I'm seeing is when trying to call new Configuration(),
> which results in:
>
>
> {code:java}
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/thirdparty/com/google/common/base/Preconditions
> at
> org.apache.hadoop.conf.Configuration$DeprecationDelta.(Configuration.java:430)
>
> at
> org.apache.hadoop.conf.Configuration$DeprecationDelta.(Configuration.java:443)
>
> at
> org.apache.hadoop.conf.Configuration.(Configuration.java:525){code}
>
>
> If you take a look at the hbase-shaded-client fat jar, it contains the
> org.apache.hadoop.conf.Configuration class as you'd expect. If you decompile
> that class (or look at the 3.3.1 source), you'll see that there is an import
> for org.apache.hadoop.thirdparty.com.google.common.base.Preconditions but the
> fat jar does not provide it.
>
> One way for clients to get around this is to add an explicit dependency on
> hadoop-shaded-guava, but this is problematic for a few reasons:
>
> - it's best practice to use maven-dependency-plugin to disallow declared,
> unused dependencies (which this would be)
> - it requires users to continually keep the version of hadoop-shaded-guava
> up-to-date over time.
> - it only covers guava, but there is also protobuf and potentially other
> shaded libraries in the future.
>
> I think we should remove the exclusion of
> {{org/apache/hadoop/thirdparty/**/*}} from the shading config and instead add
> that pattern to the allowlist so that hbase-shaded-client is all clients need
> to get started with hbase.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26760) LICENSE handling should not allow non-aggregated "apache-2.0"
Sean Busbey created HBASE-26760:
---
Summary: LICENSE handling should not allow non-aggregated
"apache-2.0"
Key: HBASE-26760
URL: https://issues.apache.org/jira/browse/HBASE-26760
Project: HBase
Issue Type: Task
Components: community
Affects Versions: 2.4.3, 2.4.10
Reporter: Sean Busbey
Fix For: 2.5.0
our non-aggregated license check currently allows the malformed name
"apache-2.0". All ALv2 licenses should be aggregated.
- remove "apache-2.0" from the {{non_aggregate_fine}} list
- figure out what dependencies are referring to this license and correct them
in {{supplemental-models.xml}}
nice to have: maybe add a precommit check that complains if the
{{non_aggregate_fine}} set is changed?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26696) Migrate our jenkins jobs from ci-hadoop to ci-hbase
[ https://issues.apache.org/jira/browse/HBASE-26696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-26696. - Resolution: Fixed this is all wrapped up now. > Migrate our jenkins jobs from ci-hadoop to ci-hbase > --- > > Key: HBASE-26696 > URL: https://issues.apache.org/jira/browse/HBASE-26696 > Project: HBase > Issue Type: Task > Components: jenkins >Reporter: Duo Zhang >Priority: Major > > We have our own jenkins controller now. > Let's do the migration. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26756) remove disabled hbase jobs from ci-hadoop
[ https://issues.apache.org/jira/browse/HBASE-26756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-26756. - Resolution: Fixed went ahead and cleared things out. > remove disabled hbase jobs from ci-hadoop > - > > Key: HBASE-26756 > URL: https://issues.apache.org/jira/browse/HBASE-26756 > Project: HBase > Issue Type: Sub-task >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > There are a bunch of disabled jobs on ci-hadoop from us cautiously moving to > ci-hbase. Delete them all once we're confident that things are moved over to > ci-hbase enough that we are going to handle future problems by fixing them > in-place on ci-hbase rather than rolling back to running on ci-hadoop. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26758) HBase Thirdparty repo should have nightly for checking main repo tests on top of SNAPSHOT artifacts
Sean Busbey created HBASE-26758:
---
Summary: HBase Thirdparty repo should have nightly for checking
main repo tests on top of SNAPSHOT artifacts
Key: HBASE-26758
URL: https://issues.apache.org/jira/browse/HBASE-26758
Project: HBase
Issue Type: Task
Components: integration tests, thirdparty
Reporter: Sean Busbey
We should have a nightly test that runs if there are changes to
* top level pom
* any of the hbase-shaded-* modules
That test should grab the current HEAD of a configurable branch of the main
repo (which defaults to the repo's default branch, currently master) and then
attempt a build + tests while setting {{hbase-thirdparty.version}} to the
in-progress SNAPSHOT build. The test run should exclude the current set of
flaky tests.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26756) remove disabled hbase jobs from ci-hadoop
Sean Busbey created HBASE-26756: --- Summary: remove disabled hbase jobs from ci-hadoop Key: HBASE-26756 URL: https://issues.apache.org/jira/browse/HBASE-26756 Project: HBase Issue Type: Sub-task Reporter: Sean Busbey There are a bunch of disabled jobs on ci-hadoop from us cautiously moving to ci-hbase. Delete them all once we're confident that things are moved over to ci-hbase enough that we are going to handle future problems by fixing them in-place on ci-hbase rather than rolling back to running on ci-hadoop. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26740) migrate ARM specific build
[
https://issues.apache.org/jira/browse/HBASE-26740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-26740.
-
Resolution: Fixed
I changed the job's label from {{arm2}} to {{arm}} after moving it to ci-hbase.
job is running now. Will check back on the results tomorrow.
> migrate ARM specific build
> --
>
> Key: HBASE-26740
> URL: https://issues.apache.org/jira/browse/HBASE-26740
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26755) HBase website generation should show error details on failure
Sean Busbey created HBASE-26755:
---
Summary: HBase website generation should show error details on
failure
Key: HBASE-26755
URL: https://issues.apache.org/jira/browse/HBASE-26755
Project: HBase
Issue Type: Bug
Components: website
Reporter: Sean Busbey
Assignee: Sean Busbey
website build failed this morning while running maven. no details because we
have to opt-in via {{--errors}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26743) migrate hbase-thirdparty PR
[ https://issues.apache.org/jira/browse/HBASE-26743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-26743. - Resolution: Fixed > migrate hbase-thirdparty PR > > > Key: HBASE-26743 > URL: https://issues.apache.org/jira/browse/HBASE-26743 > Project: HBase > Issue Type: Sub-task > Components: jenkins >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26743) migrate hbase-thirdparty PR
Sean Busbey created HBASE-26743: --- Summary: migrate hbase-thirdparty PR Key: HBASE-26743 URL: https://issues.apache.org/jira/browse/HBASE-26743 Project: HBase Issue Type: Sub-task Components: jenkins Reporter: Sean Busbey Assignee: Sean Busbey -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26740) migrate ARM specific build
Sean Busbey created HBASE-26740: --- Summary: migrate ARM specific build Key: HBASE-26740 URL: https://issues.apache.org/jira/browse/HBASE-26740 Project: HBase Issue Type: Sub-task Reporter: Sean Busbey Assignee: Sean Busbey -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26737) migrate hbase-native-client PR job
[ https://issues.apache.org/jira/browse/HBASE-26737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-26737. - Resolution: Fixed looks good. got a failure and a success w/proper notification to github. > migrate hbase-native-client PR job > -- > > Key: HBASE-26737 > URL: https://issues.apache.org/jira/browse/HBASE-26737 > Project: HBase > Issue Type: Sub-task > Components: jenkins >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26737) migrate hbase-native-client PR job
Sean Busbey created HBASE-26737: --- Summary: migrate hbase-native-client PR job Key: HBASE-26737 URL: https://issues.apache.org/jira/browse/HBASE-26737 Project: HBase Issue Type: Sub-task Components: jenkins Reporter: Sean Busbey Assignee: Sean Busbey -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26736) migrate hhbase-operator-tools PR job
[ https://issues.apache.org/jira/browse/HBASE-26736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-26736. - Resolution: Fixed > migrate hhbase-operator-tools PR job > > > Key: HBASE-26736 > URL: https://issues.apache.org/jira/browse/HBASE-26736 > Project: HBase > Issue Type: Sub-task > Components: jenkins >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26736) migrate hhbase-operator-tools PR job
Sean Busbey created HBASE-26736: --- Summary: migrate hhbase-operator-tools PR job Key: HBASE-26736 URL: https://issues.apache.org/jira/browse/HBASE-26736 Project: HBase Issue Type: Sub-task Components: jenkins Reporter: Sean Busbey Assignee: Sean Busbey -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26735) migrate hbase-connectors PR job
[ https://issues.apache.org/jira/browse/HBASE-26735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-26735. - Resolution: Fixed > migrate hbase-connectors PR job > --- > > Key: HBASE-26735 > URL: https://issues.apache.org/jira/browse/HBASE-26735 > Project: HBase > Issue Type: Sub-task > Components: jenkins >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26735) migrate hbase-connectors PR job
Sean Busbey created HBASE-26735: --- Summary: migrate hbase-connectors PR job Key: HBASE-26735 URL: https://issues.apache.org/jira/browse/HBASE-26735 Project: HBase Issue Type: Sub-task Components: jenkins Reporter: Sean Busbey Assignee: Sean Busbey -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26698) migrate website generation
[ https://issues.apache.org/jira/browse/HBASE-26698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-26698. - Resolution: Fixed > migrate website generation > -- > > Key: HBASE-26698 > URL: https://issues.apache.org/jira/browse/HBASE-26698 > Project: HBase > Issue Type: Sub-task > Components: jenkins, website >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-24936) review Jenkins build artifacts
[ https://issues.apache.org/jira/browse/HBASE-24936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-24936. - Assignee: Duo Zhang (was: Sean Busbey) Resolution: Fixed > review Jenkins build artifacts > -- > > Key: HBASE-24936 > URL: https://issues.apache.org/jira/browse/HBASE-24936 > Project: HBase > Issue Type: Task >Reporter: Sean Busbey >Assignee: Duo Zhang >Priority: Critical > Attachments: usage-hbase-find-flaky.zip, usage-hbase-full.zip > > > Post move to the ci-hadoop build servers we are now the biggest user of > space. That is not a problem in and of itself, but the master node has run > out of disk space twice now. As of this snapshot we are using 125GB of > storage and the next largest project is only using 20GB. > https://paste.apache.org/kyrds > We should review our builds for any issues and come up with expectations for > what our steady-state disk usage should look like > * we are supposed to compress any test logs (usually this gets us 90-99% > space savings) > * we are supposed to clean up workspaces when jobs are done > * we are supposed to keep a fixed window of prior builds (either by days or > number of runs) > If all of our jobs are currently following these guidelines, another > possibility is to push the artifacts we need over to > [nightlies.a.o|https://nightlies.apache.org/authoring.html]. Barring that, we > should formally request asf infra set up [a plugin for storing artifact on > s3|https://plugins.jenkins.io/artifact-manager-s3/]. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26704) EOL HBase 2.3.x
Sean Busbey created HBASE-26704: --- Summary: EOL HBase 2.3.x Key: HBASE-26704 URL: https://issues.apache.org/jira/browse/HBASE-26704 Project: HBase Issue Type: Task Components: community Reporter: Sean Busbey VOTE thread is here: https://lists.apache.org/thread/v3gfoyf255rpvql4h3g68ong5p4ps4rs * remove from https://hbase.apache.org/downloads * delete from dist svn repo https://dist.apache.org/repos/dist/release/hbase/ (which will remove the release from downloads.apache.org) * send an announcement to user@hbase and dev@hbase -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26698) migrate website generation
Sean Busbey created HBASE-26698: --- Summary: migrate website generation Key: HBASE-26698 URL: https://issues.apache.org/jira/browse/HBASE-26698 Project: HBase Issue Type: Sub-task Components: jenkins, website Reporter: Sean Busbey Assignee: Sean Busbey -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26543) HBase shell no longer properly parses several args
Sean Busbey created HBASE-26543: --- Summary: HBase shell no longer properly parses several args Key: HBASE-26543 URL: https://issues.apache.org/jira/browse/HBASE-26543 Project: HBase Issue Type: Bug Components: shell Affects Versions: 3.0.0-alpha-1, 2.5.0 Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 2.5.0, 3.0.0-alpha-2 shell cli arg parsing is broken: * debug is eating a possible arg * -D handling is consuming part of ARGV without declaring it * non-interactive mode is eating a possible arg * top level definitions is eating a possible arg * script2run is being ignored entirely -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26469) HBase shell has changed exit behavior
Sean Busbey created HBASE-26469:
---
Summary: HBase shell has changed exit behavior
Key: HBASE-26469
URL: https://issues.apache.org/jira/browse/HBASE-26469
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 2.4.8, 2.5.0, 3.0.0-alpha-2
Reporter: Sean Busbey
The HBase shell has changed behavior in a way that breaks being able to exit
properly.
Two example scripts to act as stand ins for hbase shell scripts to "do
something simple then exit":
{code}
tmp % echo "list\nexit" > clean_exit.rb
tmp % echo "list\nexit 1" > error_exit.rb
{code}
Giving these two scripts is possible:
* passed as a cli argument
* via redirected stdin
Additionally the shell invocation can be:
* in the default compatibility mode
* with the "non interactive" flag that gives an exit code that reflects runtime
errors
I'll post logs of the details as attachments but here are some tables of the
exit codes.
The {{clean_exit.rb}} invocations ought to exit with success, exit code 0.
|| || 1.4.14 || 1.7.1 || 2.0.6 || 2.1.9 || 2.2.7 || 2.3.7 || 2.4.8 ||
master ||
| cli, default |0 |0 |0 |0 |0 |0 |1 |
1* |
| cli, -n | 0 |0 |0 |0 |0 |0 |1 |
hang |
| stdin, default | 0 |0 |0 |0 |0 |0 |0 |
0|
| stdin, -n | 1 |1 |1 |1 |1 |1 |1* |
1* |
The {{error_exit.rb}} invocation should return a non-zero exit code, unless
we're specifically trying to match a normal hbase shell session.
|| || 1.4.14 || 1.7.1 || 2.0.6 || 2.1.9 || 2.2.7 || 2.3.7 || 2.4.8 ||
master ||
| cli, default | 1 |1 |1 |1 |1 |1 |1* |
1* |
| cli, -n |1 |1 |1 |1 |1 |1 |1* |
hang |
| stdin, default | 0 |0 |0 |0 |0 |0 |0 |
0|
| stdin, -n | 1 |1 |1 |1 |1 |1 |1* |
1* |
In cases marked with * the error details are different.
The biggest concern are the new-to-2.4 non-zero exit code when we should have a
success and the hanging.
The former looks like this:
{code}
ERROR NoMethodError: private method `exit' called for nil:NilClass
{code}
The change in error details for the error exit script also shows this same
detail.
This behavior appears to be a side effect of HBASE-11686. As far as I can tell,
the IRB handling of 'exit' calls fail because we implement our own handling of
sessoins rather than rely on the intended session interface. We never set a
current session, and IRB's exit implementation presumes there will be one.
Running in debug shows this in a stacktrace:
{code}
Took 0.4563 seconds
ERROR NoMethodError: private method `exit' called for nil:NilClass
NoMethodError: private method `exit' called for nil:NilClass
irb_exit at
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/extend-command.rb:30
evaluate at stdin:2
eval at org/jruby/RubyKernel.java:1048
evaluate at
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/workspace.rb:85
eval_io at uri:classloader:/shell.rb:327
each_top_level_statement at
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:246
loop at org/jruby/RubyKernel.java:1442
each_top_level_statement at
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:232
catch at org/jruby/RubyKernel.java:1189
each_top_level_statement at
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb/ruby-lex.rb:231
eval_io at uri:classloader:/shell.rb:326
classpath:/jar-bootstrap.rb at classpath:/jar-bootstrap.rb:194
exception_handler at uri:classloader:/shell.rb:339
at classpath:/jar-bootstrap.rb:194
{code}
And in our version of IRB (0.9.6) [line 30 for
extend-commands|https://github.com/ruby/irb/blob/v0.9.6/lib/irb/extend-command.rb#L30]
corresponds to this code:
{code}
# Quits the current irb context
#
# +ret+ is the optional signal or message to send to Context#exit
#
# Same as IRB.CurrentContext.exit.
def irb_exit(ret = 0)
irb_context.exit(ret)
end
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26332) Update next set of minor releases for Hadoop 3.1 EOM
Sean Busbey created HBASE-26332: --- Summary: Update next set of minor releases for Hadoop 3.1 EOM Key: HBASE-26332 URL: https://issues.apache.org/jira/browse/HBASE-26332 Project: HBase Issue Type: Task Components: hadoop3 Affects Versions: 2.5.0, 3.0.0-alpha-2 Reporter: Sean Busbey Fix For: 2.5.0, 3.0.0-alpha-2 Looks like Hadoop 3.1 went EOL back in June. I can't find a formal announcement, but "[\[VOTE\] Hadoop 3.1.x EOL|https://s.apache.org/hadoop-vote-3.1-eol]; passed and HADOOP-17759 removed 3.1 from their active downloads page and the asf active distribution mirroring. We should update our minimum testing version to 3.2 and update the hadoop dependency matrix to reflect this change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26331) verify hbase testing infra is up to date wrt H node renaming
Sean Busbey created HBASE-26331: --- Summary: verify hbase testing infra is up to date wrt H node renaming Key: HBASE-26331 URL: https://issues.apache.org/jira/browse/HBASE-26331 Project: HBase Issue Type: Task Components: test Reporter: Sean Busbey Infra is renaming all the H* nodes. https://cwiki.apache.org/confluence/display/INFRA/H+node+renaming+Proposal do a pass through the Jenkins scripts and manually configured jobs to make sure we either do not directly name H* nodes or to update such for the rename. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26031) Validate nightly builds run on new ci workers hbase11-hbase15
[ https://issues.apache.org/jira/browse/HBASE-26031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-26031. - Resolution: Later After chatting with Stack about donation internals, these hosts are getting decommed for now. > Validate nightly builds run on new ci workers hbase11-hbase15 > - > > Key: HBASE-26031 > URL: https://issues.apache.org/jira/browse/HBASE-26031 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: image-2021-06-24-16-14-03-721.png > > > Per slack, asf infra has finished adding in nodes hbase10-hbase15 to > ci-hadoop. > make sure they can run nightly. > # Set labels for all these node to "hbase-staging" > # Push a feature branch off of current HEAD that updates the agent labels to > use "hbase-staging" > # trigger a bunch of runs. make sure *something* runs on each of the nodes > # Set labels for the nodes to "hbase" > # delete feature branch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26031) Validate nightly builds run on new ci workers hbase10-hbase15
Sean Busbey created HBASE-26031: --- Summary: Validate nightly builds run on new ci workers hbase10-hbase15 Key: HBASE-26031 URL: https://issues.apache.org/jira/browse/HBASE-26031 Project: HBase Issue Type: Task Components: test Reporter: Sean Busbey Assignee: Sean Busbey Per slack, asf infra has finished adding in nodes hbase10-hbase15 to ci-hadoop. make sure they can run nightly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25555) release hbase-thirdparty 3.5.0
[ https://issues.apache.org/jira/browse/HBASE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-2. - Resolution: Fixed > release hbase-thirdparty 3.5.0 > -- > > Key: HBASE-2 > URL: https://issues.apache.org/jira/browse/HBASE-2 > Project: HBase > Issue Type: Task > Components: hbase-thirdparty >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: hbase-thirdparty-3.5.0 > > > make a release for hbase-thirdparty off of the current master branch. > Since HBASE-24802 adds a new dependency artifact we should increment the > minor version number, which gives us a proposed version of 3.5.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25589) release hbase-operator-tools 1.1.0
Sean Busbey created HBASE-25589: --- Summary: release hbase-operator-tools 1.1.0 Key: HBASE-25589 URL: https://issues.apache.org/jira/browse/HBASE-25589 Project: HBase Issue Type: Task Components: community, hbase-operator-tools Reporter: Sean Busbey Assignee: Sean Busbey -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25580) Release scripts should include in the vote email the git hash that the RC tag points to
Sean Busbey created HBASE-25580: --- Summary: Release scripts should include in the vote email the git hash that the RC tag points to Key: HBASE-25580 URL: https://issues.apache.org/jira/browse/HBASE-25580 Project: HBase Issue Type: Task Components: community Reporter: Sean Busbey Assignee: Sean Busbey the git tags for release candidates are not protected, so they could change after a RM posts a release candidate. the vote text should include the git hash that the RC tag points to at the time of the vote so a change can easily be seen. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25579) HBase Connectors pom should include nexus staging repo management
Sean Busbey created HBASE-25579:
---
Summary: HBase Connectors pom should include nexus staging repo
management
Key: HBASE-25579
URL: https://issues.apache.org/jira/browse/HBASE-25579
Project: HBase
Issue Type: Task
Reporter: Sean Busbey
The current RC creation scripts rely on project poms having a plugin configured
to handle nexus staging repository closing / dropping. The hbase-connectors
project is currently missing this plugin, which makes it so that a RM has to go
manually close the staging repository and then copy the location into the
generated vote text.
the needed plugin config is:
{code}
apache-release
org.sonatype.plugins
nexus-staging-maven-plugin
1.6.8
true
https://repository.apache.org/
apache.releases.https
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25578) HBase filesystem pom should include nexus staging repo management
Sean Busbey created HBASE-25578:
---
Summary: HBase filesystem pom should include nexus staging repo
management
Key: HBASE-25578
URL: https://issues.apache.org/jira/browse/HBASE-25578
Project: HBase
Issue Type: Task
Components: community, Filesystem Integration
Reporter: Sean Busbey
Assignee: Sean Busbey
The current RC creation scripts rely on project poms having a plugin configured
to handle nexus staging repository closing / dropping. The hbase-filesystem
project is currently missing this plugin, which makes it so that a RM has to go
manually close the staging repository and then copy the location into the
generated vote text.
the needed plugin config is:
{code}
apache-release
org.sonatype.plugins
nexus-staging-maven-plugin
1.6.8
true
https://repository.apache.org/
apache.releases.https
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25577) HBase operator tools pom should include nexus staging repo management
Sean Busbey created HBASE-25577:
---
Summary: HBase operator tools pom should include nexus staging
repo management
Key: HBASE-25577
URL: https://issues.apache.org/jira/browse/HBASE-25577
Project: HBase
Issue Type: Task
Components: community, hbase-operator-tools
Reporter: Sean Busbey
Assignee: Sean Busbey
The current RC creation scripts rely on project poms having a plugin configured
to handle nexus staging repository closing / dropping. The hbase-operator-tools
project is currently missing this plugin, which makes it so that a RM has to go
manually close the staging repository and then copy the location into the
generated vote text.
the needed plugin config is:
{code}
apache-release
org.sonatype.plugins
nexus-staging-maven-plugin
1.6.8
true
https://repository.apache.org/
apache.releases.https
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25573) release script generated vote template has incorrect staging area
Sean Busbey created HBASE-25573: --- Summary: release script generated vote template has incorrect staging area Key: HBASE-25573 URL: https://issues.apache.org/jira/browse/HBASE-25573 Project: HBase Issue Type: Bug Components: community Reporter: Sean Busbey Assignee: Sean Busbey I noticed while making the hbase-thirdparty 3.5.0 RC0 that the generated vote email showed the location of the RC using just the version and RC number: https://dist.apache.org/repos/dist/dev/hbase/3.5.0RC0/ The correct location needs to include the project name when the release is for something other than the main project, e.g.: https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-3.5.0RC0/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25572) HBase filesystem repo should have a "release" maven profile
Sean Busbey created HBASE-25572: --- Summary: HBase filesystem repo should have a "release" maven profile Key: HBASE-25572 URL: https://issues.apache.org/jira/browse/HBASE-25572 Project: HBase Issue Type: Bug Components: community Reporter: Sean Busbey Assignee: Sean Busbey Fix For: hbase-filesystem-1.0.0-alpha2 Trying to run a RC for the hbase-filesystem project fails in dryrun because there isn't a "release" profile and the RC creation scripts require one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25565) test fails on AArch64, Fedora 33, but build worked
[ https://issues.apache.org/jira/browse/HBASE-25565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-25565. - Resolution: Invalid Please use the mailing list to troubleshoot this kind of build failure. Information can be found on this page: http://hbase.apache.org/mail-lists.html If you are working on building unreleased code please use the dev@hbase list. If you are just trying to use a format release (e.g. from the downloads page) then please use user@hbase. > test fails on AArch64, Fedora 33, but build worked > -- > > Key: HBASE-25565 > URL: https://issues.apache.org/jira/browse/HBASE-25565 > Project: HBase > Issue Type: Bug >Reporter: Lutz Weischer >Priority: Major > > [jw@cn06 hbase]$ [jw@cn06 hbase]$ mvn install > [INFO] Scanning for projects... > [INFO] > > [INFO] Detecting the operating system and CPU architecture > [INFO] > > [INFO] os.detected.name: linux > [INFO] os.detected.arch: aarch_64 > [INFO] os.detected.version: 5.8 > [INFO] os.detected.version.major: 5 > [INFO] os.detected.version.minor: 8 > [INFO] os.detected.release: fedora > [INFO] os.detected.release.version: 33 > [INFO] os.detected.release.like.fedora: true > [INFO] os.detected.classifier: linux-aarch_64 > [INFO] > > [INFO] Reactor Build Order: > [INFO] > [INFO] Apache HBase > [pom] > [INFO] Apache HBase - Checkstyle > [jar] > [INFO] Apache HBase - Annotations > [jar] > [INFO] Apache HBase - Build Configuration > [pom] > [INFO] Apache HBase - Logging > [jar] > [INFO] Apache HBase - Shaded Protocol > [jar] > [INFO] Apache HBase - Common > [jar] > [INFO] Apache HBase - Metrics API > [jar] > [INFO] Apache HBase - Metrics Implementation > [jar] > [INFO] Apache HBase - Hadoop Compatibility > [jar] > [INFO] Apache HBase - Client > [jar] > [INFO] Apache HBase - Zookeeper > [jar] > [INFO] Apache HBase - Replication > [jar] > [INFO] Apache HBase - Balancer > [jar] > [INFO] Apache HBase - Resource Bundle > [jar] > [INFO] Apache HBase - HTTP > [jar] > [INFO] Apache HBase - Asynchronous FileSystem > [jar] > [INFO] Apache HBase - Procedure > [jar] > [INFO] Apache HBase - Server > [jar] > [INFO] Apache HBase - MapReduce > [jar] > [INFO] Apache HBase - Testing Util > [jar] > [INFO] Apache HBase - Thrift > [jar] > [INFO] Apache HBase - Shell > [jar] > [INFO] Apache HBase - Coprocessor Endpoint > [jar] > [INFO] Apache HBase - Backup > [jar] > [INFO] Apache HBase - Integration Tests > [jar] > [INFO] Apache HBase - Rest > [jar] > [INFO] Apache HBase - Examples > [jar] > [INFO] Apache HBase - Shaded > [pom] > [INFO] Apache HBase - Shaded - Client (with Hadoop bundled) > [jar] > [INFO] Apache HBase - Shaded - Client > [jar] > [INFO] Apache HBase - Shaded - MapReduce > [jar] > [INFO] Apache HBase - External Block Cache > [jar] > [INFO] Apache HBase - HBTop > [jar] > [INFO] Apache HBase - Assembly > [pom] > [INFO] Apache HBase - Shaded - Testing Util > [jar] > [INFO] Apache HBase - Shaded - Testing Util Tester > [jar] > [INFO] Apache HBase Shaded Packaging Invariants > [pom] > [INFO] Apache HBase Shaded Packaging Invariants (with Hadoop bundled)
[jira] [Created] (HBASE-25557) hbase thirdparty jdk bytecode enforcement shouldn't check test dependencies
Sean Busbey created HBASE-25557: --- Summary: hbase thirdparty jdk bytecode enforcement shouldn't check test dependencies Key: HBASE-25557 URL: https://issues.apache.org/jira/browse/HBASE-25557 Project: HBase Issue Type: Bug Components: hbase-thirdparty Affects Versions: hbase-thirdparty-3.5.0 Reporter: Sean Busbey Assignee: Sean Busbey Fix For: hbase-thirdparty-3.5.0 while staging a 3.5.0 RC the build failed when deploying a staged nexus repo for the hbase-noop-htrace module because the test dependency on hbase 2 fails the maven enforcer check for jdk7 compatibility. since the jdk compatibility for the artifacts we generate in hbase-thirdparty isn't impacted by test scope dependency or optional dependencies, we shouldn't check them in the enforcer rule. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25552) [hbase-thirdparty] Update jetty version to fix CVE-2020-27218
[ https://issues.apache.org/jira/browse/HBASE-25552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-25552. - Resolution: Fixed > [hbase-thirdparty] Update jetty version to fix CVE-2020-27218 > - > > Key: HBASE-25552 > URL: https://issues.apache.org/jira/browse/HBASE-25552 > Project: HBase > Issue Type: Bug > Components: dependencies, hbase-thirdparty >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Major > Fix For: hbase-thirdparty-3.5.0 > > > CVE-2020-27218 obseved in 9.4.0.RC0 to 9.4.34.v20201102, 10.0.0.alpha0 to > 10.0.0.beta2, and 11.0.0.alpha0 to 11.0.0.beta2. > Currently we are using 9.4.34.v20201102, > https://github.com/apache/hbase-thirdparty/blob/a8e3629618642fd50daa31cc027bc674ab659711/pom.xml#L139 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25530) hbase 3.0.0 example source not usable due to hbase-asyncfs-3.0.0-SNAPSHOT jar not found
[ https://issues.apache.org/jira/browse/HBASE-25530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-25530. - Resolution: Invalid this conversation moved to user@hbase in the email thread "problem building hbase 3.0.0" > hbase 3.0.0 example source not usable due to hbase-asyncfs-3.0.0-SNAPSHOT jar > not found > --- > > Key: HBASE-25530 > URL: https://issues.apache.org/jira/browse/HBASE-25530 > Project: HBase > Issue Type: Task >Reporter: Richard Tkatch >Priority: Major > > tried to run maven on the example client pom for hbase 3.0.0. > [ERROR] Failed to execute goal on project hbase-examples: Could not resolve > dependencies for project org.apache.hbase:hbase-examples:jar:3.0.0-SNAPSHOT: > The following artifacts could not be resolved: > org.apache.hbase:hbase-asyncfs:jar:3.0.0-SNAPSHOT, > org.apache.hbase:hbase-replication:jar:3.0.0-SNAPSHOT: Failure to find > org.apache.hbase:hbase-asyncfs:jar:3.0.0-SNAPSHOT in > https://repository.apache.org/snapshots was cached in the local repository, > resolution will not be reattempted until the update interval of > apache.snapshots has elapsed or updates are forced -> [Help 1] > > I can't seem to find this jar and the link that points to the jar is dead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25555) release hbase-thirdparty 3.5.0
Sean Busbey created HBASE-2: --- Summary: release hbase-thirdparty 3.5.0 Key: HBASE-2 URL: https://issues.apache.org/jira/browse/HBASE-2 Project: HBase Issue Type: Task Components: hbase-thirdparty Reporter: Sean Busbey Assignee: Sean Busbey Fix For: thirdparty-3.5.0 make a release for hbase-thirdparty off of the current master branch. Since HBASE-24802 adds a new dependency artifact we should increment the minor version number, which gives us a proposed version of 3.5.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25409) Release 2.3.4
[ https://issues.apache.org/jira/browse/HBASE-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-25409. - Resolution: Fixed > Release 2.3.4 > - > > Key: HBASE-25409 > URL: https://issues.apache.org/jira/browse/HBASE-25409 > Project: HBase > Issue Type: Task > Components: community >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25527) update download area for 2.3.4 as new stable build
[ https://issues.apache.org/jira/browse/HBASE-25527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-25527. - Resolution: Fixed > update download area for 2.3.4 as new stable build > -- > > Key: HBASE-25527 > URL: https://issues.apache.org/jira/browse/HBASE-25527 > Project: HBase > Issue Type: Sub-task > Components: community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > * update the stable symlink to point to 2.3.4 > * Remove the 2.3.3 release from downloads.a.o -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25527) update download area for 2.3.4 as new stable build
Sean Busbey created HBASE-25527: --- Summary: update download area for 2.3.4 as new stable build Key: HBASE-25527 URL: https://issues.apache.org/jira/browse/HBASE-25527 Project: HBase Issue Type: Sub-task Components: community Reporter: Sean Busbey * update the stable symlink to point to 2.3.4 * Remove the 2.3.3 release from downloads.a.o -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-25409) Release 2.3.4
[ https://issues.apache.org/jira/browse/HBASE-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reopened HBASE-25409: - > Release 2.3.4 > - > > Key: HBASE-25409 > URL: https://issues.apache.org/jira/browse/HBASE-25409 > Project: HBase > Issue Type: Task > Components: community >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25511) After hbase-thirdparty 3.4.1 release, set next dev version
Sean Busbey created HBASE-25511: --- Summary: After hbase-thirdparty 3.4.1 release, set next dev version Key: HBASE-25511 URL: https://issues.apache.org/jira/browse/HBASE-25511 Project: HBase Issue Type: Sub-task Components: hbase-thirdparty Affects Versions: thirdparty-3.5.0 Reporter: Sean Busbey Assignee: Sean Busbey Fix For: thirdparty-3.5.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25508) Add an example of using the thrift proxy in thrift-over-http mode
Sean Busbey created HBASE-25508: --- Summary: Add an example of using the thrift proxy in thrift-over-http mode Key: HBASE-25508 URL: https://issues.apache.org/jira/browse/HBASE-25508 Project: HBase Issue Type: Improvement Components: Thrift Reporter: Sean Busbey Assignee: Sean Busbey add an example that shows using the thrift proxy with over-http enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25338) Client app getting "Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane." errors
[
https://issues.apache.org/jira/browse/HBASE-25338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-25338.
-
Resolution: Incomplete
> Client app getting "Failed to load SIMPLE, KERBEROS, and DIGEST
> authentication providers. Classpath is not sane." errors
>
>
> Key: HBASE-25338
> URL: https://issues.apache.org/jira/browse/HBASE-25338
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.3.0
> Environment: Docker container build on adoptopenjdk/openjdk14:jre or
> adoptopenjdk/openjdk15:jre
> Spring Boot 2.0.1
> hbase-client 2.3.0
> hbase-server 2.3.0
>
>Reporter: ALEXEY POLYAKOV
>Priority: Major
>
> Application is ran from Spring jar, hbase-client jar is included.
> But looks like ServiceLoader.load is unable to load providers for
> SaslClientAuthenticationProvider
> I get lot of stack traces like this in log file:
> {noformat}
> 2020-11-29T00:08:37.019Z, RpcRetryingCaller{globalStartTime=1606608514899,
> pause=100, maxAttempts=6}, java.io.IOException: Call to
> hostname/xx.xx.xx.xx:16020 failed on local exception: java.io.IOException:
> java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST
> authentication providers. Classpath is not sane. at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Call to hostname/xx.xx.xx.xx:16020 failed on
> local exception: java.io.IOException: java.lang.IllegalStateException: Failed
> to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is
> not sane.
> at
> java.base/jdk.internal.reflect.GeneratedConstructorAccessor101.newInstance(Unknown
> Source)
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
> Source)
> at
> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source)
> at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
> at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:225)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:378)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:89)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:117)
> at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:132)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:422)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:316)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:89)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:572)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:45390)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:396)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:370)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> ... 4 more
> Caused by: java.io.IOException: java.lang.IllegalStateException: Failed to
> load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not
> sane.
> at org.apache.hadoop.hbase.ipc.IPCUtil.toIOE(IPCUtil.java:159)
> ... 17 more
>
[jira] [Created] (HBASE-25286) remove EOM branches from test personality file
Sean Busbey created HBASE-25286: --- Summary: remove EOM branches from test personality file Key: HBASE-25286 URL: https://issues.apache.org/jira/browse/HBASE-25286 Project: HBase Issue Type: Task Reporter: Sean Busbey while getting the PR together for HBASE-25083 I noticed our yetus personality in the default branch still has logic for branch-2.0 and branch-2.1. remove it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-20598) Upgrade to JRuby 9.2
[ https://issues.apache.org/jira/browse/HBASE-20598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-20598. - Hadoop Flags: Incompatible change Release Note: The HBase shell now relies on JRuby 9.2. This is a new major version change for JRuby. The most significant change is Ruby compatibility changed from Ruby 2.3 to Ruby 2.5. For more detailed changes please see [the JRuby release announcement for the start of the 9.2 series](https://www.jruby.org/2018/05/24/jruby-9-2-0-0.html) as well as the [general release announcement page for updates since that version](https://www.jruby.org/news). The runtime dependency versions present on the server side classpath for the Joni (now 2.1.31) and JCodings (now 1.0.55) libraries have also been updated to match those found in the JRuby version shipped with HBase. These version changes are maintenance releases and should be backwards compatible when updated in tandem. Resolution: Fixed > Upgrade to JRuby 9.2 > > > Key: HBASE-20598 > URL: https://issues.apache.org/jira/browse/HBASE-20598 > Project: HBase > Issue Type: Bug > Components: dependencies, shell >Reporter: Josh Elser >Assignee: Norbert Kalmár >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > Attachments: HBASE-20598.001.patch, HBASE-20598.002.patch > > > [~mdrob] pointed out that there's a JRuby 9.2 release. We should see if we > can get ourselves onto that from our current 9.1 release line. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25181) Add options for disabling column family encryption and choosing hash algorithm for wrapped encryption keys.
[ https://issues.apache.org/jira/browse/HBASE-25181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-25181. - Fix Version/s: 2.4.0 3.0.0-alpha-1 Release Note: This change adds options for disabling column family encryption and choosing hash algorithm for wrapped encryption keys. Changes are done such that defaults will keep the same behavior prior to this issue. Prior to this change HBase always used the MD5 hash algorithm to store a hash for encryption keys. This hash is needed to verify the secret key of the subject. (e.g. making sure that the same secrey key is used during encrypted HFile read and write). The MD5 algorithm is considered weak, and can not be used in some (e.g. FIPS compliant) clusters. Having a configurable hash enables us to use newer and more secure hash algorithms like SHA-384 or SHA-512 (which are FIPS compliant). The hash is set via the configuration option `hbase.crypto.key.hash.algorithm`. It should be set to a JDK `MessageDigest` algorithm like "MD5", "SHA-256" or "SHA-384". The default is "MD5" for backward compatibility. Alternatively, clusters which rely on an encryption at rest mechanism outside of HBase (e.g. those offered by HDFS) and wish to ensure HBase's encryption at rest system is inactive can set `hbase.crypto.enabled` to `false`. Resolution: Fixed Pushed to master and branch-2. I started backporting to branch-1, it was close but there were enough conflicts for me to step away from it. If folks would like to see it in branch-1 please post a backport PR. > Add options for disabling column family encryption and choosing hash > algorithm for wrapped encryption keys. > --- > > Key: HBASE-25181 > URL: https://issues.apache.org/jira/browse/HBASE-25181 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.3.2 >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently we are using MD5 hash algorithm to store a hash for encryption > keys. This hash is needed to verify the secret key of the subject. (e.g. > making sure that the same secret key is used during encrypted HFile / WalFile > read and write). The MD5 algorithm is considered weak, and can not be used in > some (e.g. FIPS compliant) clusters. However, currently it is not possible to > use different hash algorithm, or to disable the whole column family > encryption globally on the cluster. > In this patch: > * I introduce a backward compatible way of specifying the hash algorithm. > This enable us to use newer and/or more secure hash algorithms like SHA-384 > or SHA-512 (which are FIPS compliant). > * I added a configuration parameter to globally enable / disable the column > family encryption feature. (enabled by default for backward compatibility). > This is handy if someone wants to operate an HBase cluster making sure that > uses are only relying on other (e.g. HDFS based) encryption mechanisms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25214) about hbase introduced fasterxml‘s jackson versions and vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-25214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-25214.
-
Resolution: Duplicate
This is a duplicate of HBASE-24802. Please follow our work over there. Help in
testing out the solution would be appreciated.
> about hbase introduced fasterxml‘s jackson versions and vulnerabilities
>
>
> Key: HBASE-25214
> URL: https://issues.apache.org/jira/browse/HBASE-25214
> Project: HBase
> Issue Type: Improvement
>Reporter: openlookeng
>Priority: Blocker
>
> a lot of hbase component use htrace-core4, this htrace-core4 shaded fasterxml
> jackson(version 2.4.0)
> [INFO] | +-
> org.apache.hbase.thirdparty:hbase-shaded-miscellaneous:jar:2.2.1:compile
> [INFO] | +- org.slf4j:slf4j-api:jar:1.7.29:compile
> [INFO] | +- commons-io:commons-io:jar:2.6:compile
> [INFO] | +-
> {color:#ff}org.apache.htrace:htrace-core4:jar:4.2.0-incubating:compile{color}
> [INFO] | +- org.apache.commons:commons-crypto:jar:1.0.0:compile
> [INFO] | +-
> com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
> [INFO] | +- log4j:log4j:jar:1.2.17:compile
> [INFO] | - org.apache.yetus:audience-annotations:jar:0.5.0:compile
>
> as you known fasterxml jackson component is frequently coming out new
> vulnerabilities, like
> CVE-2016-7051、CVE-2016-3720、CVE-2018-5968、CVE-2018-11307、CVE-2018-7489、CVE-2019-14893、CVE-2019-14379、CVE-2020-14195、CVE-2020-14061、CVE-2020-8840、CVE-2019-14540、CVE-2020-10968、CVE-2020-11619、CVE-2019-17531、CVE-2019-16943、CVE-2020-14062、CVE-2020-14060、CVE-2020-1、CVE-2019-16942、CVE-2020-9546、CVE-2020-9548、CVE-2019-12384、CVE-2020-10673、CVE-2020-24750、CVE-2019-16335、CVE-2019-14439、CVE-2020-10969、CVE-2020-2、CVE-2019-12086、CVE-2019-20330、CVE-2019-17267、CVE-2020-9547、CVE-2020-3、CVE-2020-10672、CVE-2020-11620、CVE-2020-24616、CVE-2018-19362、CVE-2018-19361、CVE-2018-19360、CVE-2018-14721、CVE-2018-14720、CVE-2018-14719、CVE-2018-14718、CVE-2018-1000873、CVE-2017-7525、CVE-2017-17485、CVE-2017-15095，CVE-2019-12814
> htrace-core4 is closed 4 years ago, what about this component's
> vulnerabilities, did hbase have plan to do with this？
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25208) Running an hbase-shell command in non-interactive mode should not complain about missing files when it fails to talk to the cluster.
Sean Busbey created HBASE-25208:
---
Summary: Running an hbase-shell command in non-interactive mode
should not complain about missing files when it fails to talk to the cluster.
Key: HBASE-25208
URL: https://issues.apache.org/jira/browse/HBASE-25208
Project: HBase
Issue Type: Bug
Components: shell
Reporter: Sean Busbey
if you script out some shell commands, then run them via the shell in
non-interactive mode we sometimes give an erroneous "I couldn't find that file"
response when the master(s) are in a bad state instead of revealing the
underlying cluster issue.
e.g.
{code}
tmpfile=$(mktemp)
echo "balance_switch false" > $tmpfile
echo "exit" >> $tmpfile
$HBASE_BIN --config $CONF_DIR shell -n $tmpfile
RET=$?
rm $tmpfile
exit $RET
{code}
as an example, with a master stuck trying to initialize as active the failure
might look like
{code}
++ mktemp
+ tmpfile=/tmp/tmp.dMHTj0nVXx
+ echo balance_switch true
+ echo exit
+ bin/hbase --config /etc/hbase/conf shell -n /tmp/tmp.dMHTj0nVXx
LoadError: no such file to load -- /tmp/tmp.dMHTj0nVXx
load at org/jruby/RubyKernel.java:974
at /some/path/to/lib/hbase/bin/hirb.rb:186
+ RET=1
+ rm /tmp/tmp.dMHTj0nVXx
+ exit 1
{code}
which is very confusing. the shell should report something related to an issue
talking to the cluster rather than a problem with the passed script.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HBASE-24875) Remove the force param for unassign since it dose not take effect any more
[ https://issues.apache.org/jira/browse/HBASE-24875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reopened HBASE-24875: - > Remove the force param for unassign since it dose not take effect any more > -- > > Key: HBASE-24875 > URL: https://issues.apache.org/jira/browse/HBASE-24875 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently unassign region in fact only close it, so not need force param any > more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24875) Remove the force param for unassign since it dose not take effect any more
[ https://issues.apache.org/jira/browse/HBASE-24875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-24875. - Hadoop Flags: Incompatible change Resolution: Fixed > Remove the force param for unassign since it dose not take effect any more > -- > > Key: HBASE-24875 > URL: https://issues.apache.org/jira/browse/HBASE-24875 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently unassign region in fact only close it, so not need force param any > more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24875) Remove the force param for unassign since it dose not take effect any more
[ https://issues.apache.org/jira/browse/HBASE-24875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-24875. - Release Note: The "force" flag to various unassign commands (java api, shell, etc) has been ignored since HBase 2. As of this change the methods that take it are now deprecated. Downstream users should stop passing/using this flag. The Admin and AsyncAdmin Java APIs will have the deprecated version of the unassign method with a force flag removed in HBase 4. Callers can safely continue to use the deprecated API until then; the internal implementation just calls the new method. The MasterObserver coprocessor API deprecates the `preUnassign` and `postUnassign` methods that include the force parameter and replaces them with versions that omit this parameter. The deprecated methods will be removed from the API in HBase 3. Until then downstream coprocessor implementations can safely continue to *just* implement the deprecated method if they wish; the replacement methods provide a default implementation that calls the deprecated method with force set to `false`. Resolution: Fixed > Remove the force param for unassign since it dose not take effect any more > -- > > Key: HBASE-24875 > URL: https://issues.apache.org/jira/browse/HBASE-24875 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently unassign region in fact only close it, so not need force param any > more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-24875) Remove the force param for unassign since it dose not take effect any more
[ https://issues.apache.org/jira/browse/HBASE-24875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reopened HBASE-24875: - > Remove the force param for unassign since it dose not take effect any more > -- > > Key: HBASE-24875 > URL: https://issues.apache.org/jira/browse/HBASE-24875 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently unassign region in fact only close it, so not need force param any > more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25182) Validate earlier coprocessors against HBase 3.0+
Sean Busbey created HBASE-25182: --- Summary: Validate earlier coprocessors against HBase 3.0+ Key: HBASE-25182 URL: https://issues.apache.org/jira/browse/HBASE-25182 Project: HBase Issue Type: Task Components: tooling Affects Versions: 3.0.0-alpha-1 Reporter: Sean Busbey we should do something similar to HBASE-20656 but for the coprocessor APIs folks may be using from hbase 2. Probably a good idea to make sure the check against HBase 1 coprocessors still works, presuming we still support any HBase 1 by the time we expect HBase 3. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25141) ref guide for 2.2.z builds needs to be updated to include 2.2 specific details
Sean Busbey created HBASE-25141: --- Summary: ref guide for 2.2.z builds needs to be updated to include 2.2 specific details Key: HBASE-25141 URL: https://issues.apache.org/jira/browse/HBASE-25141 Project: HBase Issue Type: Task Components: documentation, website Reporter: Sean Busbey Fix For: 2.2.7 for example, it's currently missing a column for 2.2 in the hadoop compat matrix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25083) make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version
Sean Busbey created HBASE-25083: --- Summary: make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version Key: HBASE-25083 URL: https://issues.apache.org/jira/browse/HBASE-25083 Project: HBase Issue Type: Task Components: documentation, hadoop2 Reporter: Sean Busbey Fix For: 1.7.0 Our reference guide list of prerequisites still has Hadoop 2.8 and 2.9 listed for HBase 1 releases. * [hadoop 2.8 is EOM|https://lists.apache.org/thread.html/r348f7bc93a522f05b7cce78a911854d128a6b1b8bd8124bad4d06ce6%40%3Cuser.hadoop.apache.org%3E] * [hadoop 2.9 is EOM|https://lists.apache.org/thread.html/r16b14cce9504f7a9d228612c6b808e72d8dd20863c78be51a7e04ed5%40%3Cuser.hadoop.apache.org%3E] The current list in the reference guide for HBase 1.6 is just the 1.5 list copied. we should update it to remove 2.8 and 2.9 and make sure we're no longer doing build/test based on those versions for branch-1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24875) Remove the force param for unassign since it dose not take effect any more
[ https://issues.apache.org/jira/browse/HBASE-24875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-24875. - Fix Version/s: 3.0.0-alpha-1 Resolution: Fixed > Remove the force param for unassign since it dose not take effect any more > -- > > Key: HBASE-24875 > URL: https://issues.apache.org/jira/browse/HBASE-24875 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > Currently unassign region in fact only close it, so not need force param any > more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25019) check if we're impacted by MASSEMBLY-941 and mitigate if needed
Sean Busbey created HBASE-25019: --- Summary: check if we're impacted by MASSEMBLY-941 and mitigate if needed Key: HBASE-25019 URL: https://issues.apache.org/jira/browse/HBASE-25019 Project: HBase Issue Type: Task Components: scripts Affects Versions: 1.6.0, 2.3.1, 2.3.0, 3.0.0-alpha-1, 1.7.0 Reporter: Sean Busbey Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0 MASSEMBLY-941 notes a bug starting in version 3.2.0 of the assembly plugin where scripts lose their executable bit. We've had this version since updating to the apache parent pom version 22 in HBASE-23675. We should check our release artifacts to see if we are impacted and if so downgrade the assembly plugin in our poms to 3.1.1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25018) EOM cleanup
Sean Busbey created HBASE-25018: --- Summary: EOM cleanup Key: HBASE-25018 URL: https://issues.apache.org/jira/browse/HBASE-25018 Project: HBase Issue Type: Task Components: community, website Reporter: Sean Busbey Assignee: Sean Busbey the [foundation downloads area for the project|https://downloads.apache.org/hbase/] has some versions that shouldn't be present anymore. Also it's missing an EOM marker for 1.3. the ref guide also includes several EOM versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24936) review Jenkins build artifacts
Sean Busbey created HBASE-24936: --- Summary: review Jenkins build artifacts Key: HBASE-24936 URL: https://issues.apache.org/jira/browse/HBASE-24936 Project: HBase Issue Type: Task Reporter: Sean Busbey Assignee: Sean Busbey Post move to the ci-hadoop build servers we are now the biggest user of space. That is not a problem in and of itself, but the master node has run out of disk space twice now. As of this snapshot we are using 125GB of storage and the next largest project is only using 20GB. https://paste.apache.org/kyrds We should review our builds for any issues and come up with expectations for what our steady-state disk usage should look like * we are supposed to compress any test logs (usually this gets us 90-99% space savings) * we are supposed to clean up workspaces when jobs are done * we are supposed to keep a fixed window of prior builds (either by days or number of runs) If all of our jobs are currently following these guidelines, another possibility is to push the artifacts we need over to [nightlies.a.o|https://nightlies.apache.org/authoring.html]. Barring that, we should formally request asf infra set up [a plugin for storing artifact on s3|https://plugins.jenkins.io/artifact-manager-s3/]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24869) migrate website generation to new asf jenkins
Sean Busbey created HBASE-24869: --- Summary: migrate website generation to new asf jenkins Key: HBASE-24869 URL: https://issues.apache.org/jira/browse/HBASE-24869 Project: HBase Issue Type: Task Components: build, website Reporter: Sean Busbey Assignee: Sean Busbey Update our website generation so we can use it on the new jenkins ci server * needs a job name that has no spaces (or fix the script to handle paths with spaces) * needs to only run on nodes labeled git-websites (so it will have the creds to push updates) * needs to set editable email notification on failure (details in comment) Also we will need to converte to a pipeline DSL * define tools, namely maven (alternative get [Tool Environment Plugin|https://plugins.jenkins.io/toolenv/] * set timeout for 4 hours (alternative get [build timeout plugin|https://plugins.jenkins.io/build-timeout/] * needs to clean worksapce when done (haven't found an alternatiave, maybe it's a default for non-pipeline jobs now?) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-24788) Fix the connection leaks on getting hbase admin from unclosed connection
[
https://issues.apache.org/jira/browse/HBASE-24788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reopened HBASE-24788:
-
this change broke source compatibility by removing the throws for
{{InterruptedException}} from the {{getRecordWriter}} and {{checkOutputSpecs}}
methods of {{TableOutputFormat}}.
I don't see any discussion of doing so, so I am guessing it was an oversight.
Normally we don't allow these kinds of changes in maintenance or minor releases.
[~te...@apache.org], [~vjasani], and [~bharathv]: please fix for at least
branch-1, branch-2, branch-2.3, and branch-2.2. If you want to keep the removal
for master please flag the change as incompatible and release note it
apparently.
> Fix the connection leaks on getting hbase admin from unclosed connection
>
>
> Key: HBASE-24788
> URL: https://issues.apache.org/jira/browse/HBASE-24788
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.6.0
>Reporter: Sandeep Pal
>Assignee: Sandeep Pal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7
>
>
> Observed the significant increase in ZK connection on performance testing on
> map reduce jobs. Turns out the
> [TableOutputFormat.checkOutputSpecs()|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java#L182]
> is not closing the connection it uses to get the hbase admin. It closes the
> hbase admin but never close the connection to get the admin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24476) release scripts should provide timing information
[ https://issues.apache.org/jira/browse/HBASE-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-24476. - Fix Version/s: 3.0.0-alpha-1 Resolution: Fixed > release scripts should provide timing information > - > > Key: HBASE-24476 > URL: https://issues.apache.org/jira/browse/HBASE-24476 > Project: HBase > Issue Type: Improvement >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Minor > Fix For: 3.0.0-alpha-1 > > > right now I can get timing from the individual maven commands but it would be > nice to get higher level times. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24572) release scripts should try to use a keyid when refering to GPG keys.
[
https://issues.apache.org/jira/browse/HBASE-24572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-24572.
-
Fix Version/s: 3.0.0-alpha-1
Resolution: Fixed
> release scripts should try to use a keyid when refering to GPG keys.
>
>
> Key: HBASE-24572
> URL: https://issues.apache.org/jira/browse/HBASE-24572
> Project: HBase
> Issue Type: Task
> Components: build, community
>Reporter: Nick Dimiduk
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Right now the template us substituting the release manager's email address in
> for the variable {{GPG_KEY}}. I think it doesn't hurt to make note of the
> email address, but what we really want here is the key's fingerprint, or some
> meaningly identifiable portion of it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24805) HBaseTestingUtility.getConnection should be threadsafe
Sean Busbey created HBASE-24805:
---
Summary: HBaseTestingUtility.getConnection should be threadsafe
Key: HBASE-24805
URL: https://issues.apache.org/jira/browse/HBASE-24805
Project: HBase
Issue Type: Bug
Components: test
Reporter: Sean Busbey
the current javadoc for getConnection carries a thread safety warning:
{code}
/**
* Get a Connection to the cluster. Not thread-safe (This class needs a lot
of work to make it
* thread-safe).
* @return A Connection that can be shared. Don't close. Will be closed on
shutdown of cluster.
*/
public Connection getConnection() throws IOException {
{code}
We then ignore that warning across our test base. We should make the method
threadsafe since the intention is to share a single Connection across all users
of the HTU instance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24794) hbase.rowlock.wait.duration should not be <= 0
Sean Busbey created HBASE-24794:
---
Summary: hbase.rowlock.wait.duration should not be <= 0
Key: HBASE-24794
URL: https://issues.apache.org/jira/browse/HBASE-24794
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.6.0, 2.3.0, 2.2.0, 1.5.0, 1.4.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 1.4.14, 2.2.6
had a cluster fail after upgrade from hbase 1 because all writes to meta failed.
RS with meta looks like:
{code}
2020-07-28 17:52:56,553 WARN org.apache.hadoop.hbase.regionserver.HRegion:
Failed getting lock, row=some_user_table
java.io.IOException: Timed out waiting for lock for row: some_user_table in
region 1588230740
at
org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5863)
at
org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.lockRowsAndBuildMiniBatch(HRegion.java:3322)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4018)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3992)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3923)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3914)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3928)
at
org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4255)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3047)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2827)
at
org.apache.hadoop.hbase.client.ClientServiceCallable.doMutate(ClientServiceCallable.java:55)
at org.apache.hadoop.hbase.client.HTable$3.rpcCall(HTable.java:538)
at org.apache.hadoop.hbase.client.HTable$3.rpcCall(HTable.java:533)
at
org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:542)
at
org.apache.hadoop.hbase.MetaTableAccessor.put(MetaTableAccessor.java:1339)
at
org.apache.hadoop.hbase.MetaTableAccessor.putToMetaTable(MetaTableAccessor.java:1329)
at
org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1672)
at
org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1112)
at
org.apache.hadoop.hbase.master.TableStateManager.fixTableStates(TableStateManager.java:296)
at
org.apache.hadoop.hbase.master.TableStateManager.start(TableStateManager.java:269)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1004)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2274)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:583)
at java.lang.Thread.run(Thread.java:745)
{code}
logging roughly 6k times /second.
failure was caused by a change in behavior for {{hbase.rowlock.wait.duration}}
in HBASE-17210 (so 1.4.0+, 2.0.0+). Prior to that change setting the config <=
0 meant that row locks would succeed only if they were immediately available.
After the change we fail the lock attempt without checking the lock at all.
workaround: set {{hbase.rowlock.wait.duration}} to a small positive number,
e.g. 1, if you want row locks to fail quickly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24773) release scripts should check for tools it will at the start of building
Sean Busbey created HBASE-24773:
---
Summary: release scripts should check for tools it will at the
start of building
Key: HBASE-24773
URL: https://issues.apache.org/jira/browse/HBASE-24773
Project: HBase
Issue Type: Bug
Reporter: Sean Busbey
I just had a RC build fail *after* going through all the work to build
everything because a recent OS upgrade meant that my {{svn}} command had been
replaced by a placeholder that provided an error message about how X-Code is no
longer providing subversion binaries.
getting a new copy of svn installed was easy enough, but if we're going to use
subversion we should have an init check similar to what we have for yetus /
maven / etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24667) Rename config that disables reverse DNS to put it in hbase.unsafe
Sean Busbey created HBASE-24667:
---
Summary: Rename config that disables reverse DNS to put it in
hbase.unsafe
Key: HBASE-24667
URL: https://issues.apache.org/jira/browse/HBASE-24667
Project: HBase
Issue Type: Task
Components: conf, Operability
Reporter: Sean Busbey
HBASE-18226 added a config for disabling reverse DNS checks
{{hbase.regionserver.hostname.disable.master.reversedns}} and the release note
calls out that the config is dangerous:
{quote}
he following config is added by this JIRA:
hbase.regionserver.hostname.disable.master.reversedns
This config is for experts: don't set its value unless you really know what you
are doing.
When set to true, regionserver will use the current node hostname for the
servername and HMaster will skip reverse DNS lookup and use the hostname sent
by regionserver instead. Note that this config and hbase.regionserver.hostname
are mutually exclusive. See https://issues.apache.org/jira/browse/HBASE-18226
for more details.
Caution: please make sure rolling upgrade succeeds before turning on this
feature.
{quote}
We should make clear the risks of using this config by making sure the name
starts with {{hbase.unsafe}}.
Rename hbase.regionserver.hostname.disable.master.reversedns}} to
{{}}{{hbase.unsafe.regionserver.hostname.disable.master.reversedns}}{{}} but
make sure the former is kept with a "deprecated config name" warning.}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24651) release script utils should set local user when GPG_KEY is defined.
Sean Busbey created HBASE-24651: --- Summary: release script utils should set local user when GPG_KEY is defined. Key: HBASE-24651 URL: https://issues.apache.org/jira/browse/HBASE-24651 Project: HBase Issue Type: Bug Components: build, community Reporter: Sean Busbey Assignee: Sean Busbey right now the release scripts fail to set the gpg local user to the specified gpg key during the actual build steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23339) Release scripts should not need to write out a copy of gpg key material
[
https://issues.apache.org/jira/browse/HBASE-23339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-23339.
-
Fix Version/s: 3.0.0-alpha-1
Resolution: Fixed
> Release scripts should not need to write out a copy of gpg key material
> ---
>
> Key: HBASE-23339
> URL: https://issues.apache.org/jira/browse/HBASE-23339
> Project: HBase
> Issue Type: Improvement
> Components: community, scripts
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0-alpha-1
>
>
> We shouldn't need to export a copy of the gpg key. Either we should give the
> container access to the gpg-agent on the host or we should do signing steps
> outside of the container.
> I don't think the latter is feasible, since the build with {{-P
> apache-release}} needs access to keys to sign as a part of the build.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24476) release scripts should provide timing information
Sean Busbey created HBASE-24476: --- Summary: release scripts should provide timing information Key: HBASE-24476 URL: https://issues.apache.org/jira/browse/HBASE-24476 Project: HBase Issue Type: Improvement Reporter: Sean Busbey Assignee: Sean Busbey right now I can get timing from the individual maven commands but it would be nice to get higher level times. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24466) Release scripts in docker mode should be able to use a named volume for maven repo
Sean Busbey created HBASE-24466: --- Summary: Release scripts in docker mode should be able to use a named volume for maven repo Key: HBASE-24466 URL: https://issues.apache.org/jira/browse/HBASE-24466 Project: HBase Issue Type: Improvement Components: build, community Reporter: Sean Busbey Assignee: Sean Busbey Allow optional volume name to use to mount the maven repo so we get reuse of third party artifacts between runs. Forcefully remove artifacts we build from it (assuming we can't make a read-only one that's shared for third party dependencies and an isolated one that's for what we're building) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24461) set up reproducible builds
Sean Busbey created HBASE-24461: --- Summary: set up reproducible builds Key: HBASE-24461 URL: https://issues.apache.org/jira/browse/HBASE-24461 Project: HBase Issue Type: Wish Components: build, community Reporter: Sean Busbey it would be really nice if we had reproducible builds https://maven.apache.org/guides/mini/guide-reproducible-builds.html if we had reproducible builds it'd be easier to e.g. do piecemeal optimization of our release process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24458) release scripts using docker on mac os should specify needed disk consistency
Sean Busbey created HBASE-24458: --- Summary: release scripts using docker on mac os should specify needed disk consistency Key: HBASE-24458 URL: https://issues.apache.org/jira/browse/HBASE-24458 Project: HBase Issue Type: Improvement Reporter: Sean Busbey Assignee: Sean Busbey reading through existing docs and write ups for Docker Desktop (the dominant docker option for running on OS X machines), it looks like the host-to-container mapping get significant perf gains if we can relax from strict container-host consistency on the filesystem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24457) release scripts on mac os are too slow
Sean Busbey created HBASE-24457: --- Summary: release scripts on mac os are too slow Key: HBASE-24457 URL: https://issues.apache.org/jira/browse/HBASE-24457 Project: HBase Issue Type: Umbrella Components: community Reporter: Sean Busbey Assignee: Sean Busbey related changes to try to bring the time down to something I can use -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24063) Make branch-2.1 EOM
[ https://issues.apache.org/jira/browse/HBASE-24063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-24063. - Resolution: Fixed > Make branch-2.1 EOM > --- > > Key: HBASE-24063 > URL: https://issues.apache.org/jira/browse/HBASE-24063 > Project: HBase > Issue Type: Task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
