[jira] [Resolved] (HBASE-24785) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.0
[ https://issues.apache.org/jira/browse/HBASE-24785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-24785. --- Resolution: Fixed Merged the addendum to master. > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.0 > > > Key: HBASE-24785 > URL: https://issues.apache.org/jira/browse/HBASE-24785 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.4.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-24785) [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.0
[ https://issues.apache.org/jira/browse/HBASE-24785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-24785: --- > [hbase-thirdparty] Generate CHANGES.md and RELEASENOTES.md for 3.4.0 > > > Key: HBASE-24785 > URL: https://issues.apache.org/jira/browse/HBASE-24785 > Project: HBase > Issue Type: Sub-task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.4.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24804) Follow up work add client side scan metrics for read replica
Huaxiang Sun created HBASE-24804: Summary: Follow up work add client side scan metrics for read replica Key: HBASE-24804 URL: https://issues.apache.org/jira/browse/HBASE-24804 Project: HBase Issue Type: New Feature Components: read replicas Affects Versions: 2.4.0 Reporter: Huaxiang Sun Assignee: Huaxiang Sun This is a followup work for HBASE-18436, which adds client metrics for read replica get. Will add metrics for scan as well. This metrics will be used in PE and any interested applications. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24798) [hbase-thirdparty] Ignore module-info and SubmissionPublisher in enforceBytecodeVersion rule
[
https://issues.apache.org/jira/browse/HBASE-24798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang resolved HBASE-24798.
---
Hadoop Flags: Reviewed
Release Note:
Ignore module-info in enforceBytecodeVersion rule as it is for supporting for
JDK9+ and will not break JDK8 support.
Ignore SubmissionPublisher in enforceBytecodeVersion rule as it is used by
jersey-common for supporting JDK11 and will not break JDK8 support.
Jersey 2.x promises to support JDK8.
Resolution: Fixed
Merged to master.
Thanks [~busbey] for reviewing.
> [hbase-thirdparty] Ignore module-info and SubmissionPublisher in
> enforceBytecodeVersion rule
>
>
> Key: HBASE-24798
> URL: https://issues.apache.org/jira/browse/HBASE-24798
> Project: HBase
> Issue Type: Bug
> Components: build, thirdparty
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: thirdparty-3.4.0
>
>
> Got this when releasing thirdparty-3.4.0.
> {noformat}
> 10:46:33 [INFO] --- maven-enforcer-plugin:1.4.1:enforce
> (enforce-maven-version) @ hbase-shaded-jersey ---
> 10:46:33 [INFO] Restricted to JDK 1.8 yet
> org.glassfish.jersey.core:jersey-common:jar:2.31:compile contains
> META-INF/versions/11/org/glassfish/jersey/internal/jsr166/SubmissionPublisher$1.class
> targeted to 55.0
> 10:46:33 [INFO] Restricted to JDK 1.8 yet
> jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6:compile contains module-info.class
> targeted to JDK 1.9
> 10:46:33 [WARNING] Rule 1:
> org.apache.maven.plugins.enforcer.EnforceBytecodeVersion failed with message:
> HBase has unsupported dependencies.
> HBase requires that all dependencies be compiled with version 1.8 or earlier
> of the JDK to properly build from source. You appear to be using a newer
> dependency. You can use
> either "mvn -version" or "mvn enforcer:display-info" to verify what version
> is active.
> Non-release builds can temporarily build with a newer JDK version by
> setting the
> 'compileSource' property (eg. mvn -DcompileSource=1.8 clean package).
> Found Banned Dependency: org.glassfish.jersey.core:jersey-common:jar:2.31
> Found Banned Dependency: jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24801) [hbase-thirdparty] Remove assembly plugin for generating src tarballs
[ https://issues.apache.org/jira/browse/HBASE-24801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-24801. --- Hadoop Flags: Reviewed Release Note: Remove the maven-assembly-plugin for generating src tarballs. We will generate it with 'git archive' command in release scripts. Resolution: Fixed Merged to master. Thanks [~stack] and [~janh] for reviewing. > [hbase-thirdparty] Remove assembly plugin for generating src tarballs > - > > Key: HBASE-24801 > URL: https://issues.apache.org/jira/browse/HBASE-24801 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: thirdparty-3.4.0 > > > As we will use git archive to generate the src tarballs in release script, we > do not need to use assembly any more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24803) Unify hbase-shell ::Shell::Commands::Command#help behavior
Elliot Miller created HBASE-24803: - Summary: Unify hbase-shell ::Shell::Commands::Command#help behavior Key: HBASE-24803 URL: https://issues.apache.org/jira/browse/HBASE-24803 Project: HBase Issue Type: Improvement Components: shell Affects Versions: 2.3.0 Reporter: Elliot Miller Assignee: Elliot Miller Fix For: 3.0.0-alpha-1, 2.3.1 This is a small patch to unify shell help: * Many of the commands use different indentation levels to style the help text. * A few commands printed help instead of returning it, which breaks our pattern. I also added a test to make sure the help text stays consistent after this patch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24802) Please fix CVEs by removing reference to htrace-core4
Rodney Aaron Stainback created HBASE-24802: -- Summary: Please fix CVEs by removing reference to htrace-core4 Key: HBASE-24802 URL: https://issues.apache.org/jira/browse/HBASE-24802 Project: HBase Issue Type: Bug Components: Client Affects Versions: 2.3.0 Reporter: Rodney Aaron Stainback htrace-core4 is a retired project and even on the latest version they Shade Jackson databind version 2.4.0 which has the following CVEs: |cve|severity|cvss| |CVE-2017-15095|critical|9.8| |CVE-2018-1000873|medium|6.5| |CVE-2018-14718|critical|9.8| |CVE-2018-5968|high|8.1| |CVE-2018-7489|critical|9.8| |CVE-2019-14540|critical|9.8| |CVE-2019-14893|critical|9.8| |CVE-2019-16335|critical|9.8| |CVE-2019-16942|critical|9.8| |CVE-2019-16943|critical|9.8| |CVE-2019-17267|critical|9.8| |CVE-2019-17531|critical|9.8| |CVE-2019-20330|critical|9.8| |CVE-2020-10672|high|8.8| |CVE-2020-10673|high|8.8| |CVE-2020-10968|high|8.8| |CVE-2020-10969|high|8.8| |CVE-2020-1|high|8.8| |CVE-2020-2|high|8.8| |CVE-2020-3|high|8.8| |CVE-2020-11619|critical|9.8| |CVE-2020-11620|critical|9.8| |CVE-2020-14060|high|8.1| |CVE-2020-14061|high|8.1| |CVE-2020-14062|high|8.1| |CVE-2020-14195|high|8.1| |CVE-2020-8840|critical|9.8| |CVE-2020-9546|critical|9.8| |CVE-2020-9547|critical|9.8| |CVE-2020-9548|critical|9.8| -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-11686) Shell code should create a binding / irb workspace instead of polluting the root namespace
[
https://issues.apache.org/jira/browse/HBASE-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliot Miller reopened HBASE-11686:
---
> Shell code should create a binding / irb workspace instead of polluting the
> root namespace
> --
>
> Key: HBASE-11686
> URL: https://issues.apache.org/jira/browse/HBASE-11686
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Sean Busbey
>Assignee: Elliot Miller
>Priority: Minor
> Fix For: 3.0.0-alpha-1
>
>
> Right now, the shell builds a list of commands and then injects them into the
> root exectution's context
> bin/hirb.rb
> {code}
> # Add commands to this namespace
> @shell.export_commands(self)
> {code}
> hbase-shell/src/main/ruby/shell.rb
> {code}
> def export_commands(where)
> ::Shell.commands.keys.each do |cmd|
> # here where is the IRB namespace
> # this method just adds the call to the specified command
> # which just references back to 'this' shell object
> # a decently extensible way to add commands
> where.send :instance_eval, <<-EOF
> def #{cmd}(*args)
> ret = @shell.command('#{cmd}', *args)
> puts
> return ret
> end
> EOF
> end
> end
> {code}
> This is an unclean abstraction. For one, it requires that there be an
> instance variable in the main namespace called '@shell' without making that
> clear in the docs. Additionally, it complicates maintenance by breaking
> isolation.
> We should update things so that shell can provide a binding for eval or a
> workspace for IRB execution and then use it directly when we construct our
> IRB session.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24788) Fix the connection leaks on getting hbase admin from unclosed connection
[ https://issues.apache.org/jira/browse/HBASE-24788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada resolved HBASE-24788. -- Resolution: Fixed Thanks Sandeep for the fix. > Fix the connection leaks on getting hbase admin from unclosed connection > > > Key: HBASE-24788 > URL: https://issues.apache.org/jira/browse/HBASE-24788 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.6.0 >Reporter: Sandeep Pal >Assignee: Sandeep Pal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7 > > > Observed the significant increase in ZK connection on performance testing on > map reduce jobs. Turns out the > [TableOutputFormat.checkOutputSpecs()|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java#L182] > is not closing the connection it uses to get the hbase admin. It closes the > hbase admin but never close the connection to get the admin. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24801) [hbase-thirdparty] Remove assembly plugin for generating src tarballs
Duo Zhang created HBASE-24801: - Summary: [hbase-thirdparty] Remove assembly plugin for generating src tarballs Key: HBASE-24801 URL: https://issues.apache.org/jira/browse/HBASE-24801 Project: HBase Issue Type: Task Components: thirdparty Reporter: Duo Zhang Assignee: Duo Zhang Fix For: thirdparty-3.4.0 As we will use git archive to generate the src tarballs in release script, we do not need to use assembly any more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24800) Enhance ACL region initialization
Pankaj Kumar created HBASE-24800:
Summary: Enhance ACL region initialization
Key: HBASE-24800
URL: https://issues.apache.org/jira/browse/HBASE-24800
Project: HBase
Issue Type: Improvement
Components: acl, MTTR
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
RegionServer persist ACL table entries into Zookeeper during ACL region open,
{code}
private void initialize(RegionCoprocessorEnvironment e) throws
IOException {
final Region region = e.getRegion();
Configuration conf = e.getConfiguration();
Map> tables =
PermissionStorage.loadAll(region);
// For each table, write out the table's permissions to the respective
// znode for that table.
for (Map.Entry> t:
tables.entrySet()) {
byte[] entry = t.getKey();
ListMultimap perms = t.getValue();
byte[] serialized = PermissionStorage.writePermissionsAsBytes(perms,
conf);
zkPermissionWatcher.writeToZookeeper(entry, serialized);
}
initialized = true;
}
{code}
Currently RegionServer send 2 RPC (one to create the table path and another to
set the data) for each table sequentially.
{code}
try {
ZKUtil.createWithParents(watcher, zkNode);
ZKUtil.updateExistingNodeData(watcher, zkNode, permsData, -1);
} catch (KeeperException e) {
LOG.error("Failed updating permissions for entry '" +
entryName + "'", e);
watcher.abort("Failed writing node "+zkNode+" to zookeeper", e);
}
{code}
If a cluster have huge number of tables then ACL region open will take time.
Example, it took ~9 min to write 60k tables ACL into ZK.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24799) Do not use install and assembly together in release scripts
Duo Zhang created HBASE-24799: - Summary: Do not use install and assembly together in release scripts Key: HBASE-24799 URL: https://issues.apache.org/jira/browse/HBASE-24799 Project: HBase Issue Type: Bug Components: scripts Reporter: Duo Zhang Assignee: Duo Zhang It will easy to introduce strang error, when releasing thirdparty 3.4.0, I hit duplicated services file when shading... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24798) [hbase-thirdparty] Exclude module-info.class in enforceBytecodeVersion rule
Duo Zhang created HBASE-24798:
-
Summary: [hbase-thirdparty] Exclude module-info.class in
enforceBytecodeVersion rule
Key: HBASE-24798
URL: https://issues.apache.org/jira/browse/HBASE-24798
Project: HBase
Issue Type: Bug
Components: build, thirdparty
Reporter: Duo Zhang
Assignee: Duo Zhang
Fix For: thirdparty-3.4.0
Got this when releasing thirdparty-3.4.0.
{noformat}
10:46:33 [INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven-version)
@ hbase-shaded-jersey ---
10:46:33 [INFO] Restricted to JDK 1.8 yet
org.glassfish.jersey.core:jersey-common:jar:2.31:compile contains
META-INF/versions/11/org/glassfish/jersey/internal/jsr166/SubmissionPublisher$1.class
targeted to 55.0
10:46:33 [INFO] Restricted to JDK 1.8 yet
jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6:compile contains module-info.class
targeted to JDK 1.9
10:46:33 [WARNING] Rule 1:
org.apache.maven.plugins.enforcer.EnforceBytecodeVersion failed with message:
HBase has unsupported dependencies.
HBase requires that all dependencies be compiled with version 1.8 or earlier
of the JDK to properly build from source. You appear to be using a newer
dependency. You can use
either "mvn -version" or "mvn enforcer:display-info" to verify what version
is active.
Non-release builds can temporarily build with a newer JDK version by setting
the
'compileSource' property (eg. mvn -DcompileSource=1.8 clean package).
Found Banned Dependency: org.glassfish.jersey.core:jersey-common:jar:2.31
Found Banned Dependency: jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6
Use 'mvn dependency:tree' to locate the source of the banned dependencies.
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24797) Move log code out of loop
[
https://issues.apache.org/jira/browse/HBASE-24797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani resolved HBASE-24797.
--
Fix Version/s: 2.4.0
2.3.1
Hadoop Flags: Reviewed
Resolution: Fixed
> Move log code out of loop
> -
>
> Key: HBASE-24797
> URL: https://issues.apache.org/jira/browse/HBASE-24797
> Project: HBase
> Issue Type: Bug
> Components: Normalizer
>Affects Versions: 3.0.0-alpha-1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>
> In HMaster#normalizeRegions, maybe we shoule move the log code about
> submittedPlanProcIds out of loop.
>
> {code:java}
> public boolean normalizeRegions() throws IOException {
> ...
> final List submittedPlanProcIds = new ArrayList<>();
> for (TableName table : allEnabledTables) {
> ...
> for (NormalizationPlan plan : plans) {
> long procId = plan.submit(this);
> submittedPlanProcIds.add(procId);
> ...
> }
> int totalPlansSubmitted = submittedPlanProcIds.size();
> if (totalPlansSubmitted > 0 && LOG.isDebugEnabled()) {
> LOG.debug("Normalizer plans submitted. Total plans count: {} , procID
> list: {}",
> totalPlansSubmitted, submittedPlanProcIds);
> }
> }
> ...
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] we need to take action if we want asf jenkins managed tests after Aug 15 2020.
https://issues.apache.org/jira/browse/INFRA-20613 > On 2020. Jul 30., at 1:47, 张铎(Duo Zhang) wrote: > > This never worked in the past... > > But it would be great if you can kick the infra team to get this done :) > > File an infra issue? > > Andor Molnar 于2020年7月29日 周三18:36写道： > >> You’re having the same issue with HBase Robot btw. At the end of console >> outputs: >> >> "Could not update commit status, please check if your scan credentials >> belong to a member of the organization or a collaborator of the repository >> and repo:status scope is selected” >> >> ...and shortly after that: >> >> "GitHub has been notified of this commit’s build result” >> >> Whatever does it mean. >> >> Andor >> >> >> >>> On 2020. Jul 29., at 11:57, Andor Molnar wrote: >>> >>> Yep, we’ve finally received it. It’s done. >>> >>> Current issue is that Jenkins is unable to set Github build status. I’ve >> added repo:status permission, but it’s also asking to be member of the >> project/organization and not sure how to do that. >>> >>> Andor >>> >>> >>> On 2020. Jul 29., at 4:10, 张铎(Duo Zhang) wrote: Seems you have already made it? Usually there are several moderators for the private list, you need to >> ask them to let the GitHub registration go through. Andor Molnar 于2020年7月29日周三 上午1:03写道： > Thanks Duo, that’s very helpful. > I cannot set private@zookeeper as a verified e-mail address, because >> the > verification e-mail cannot be sent to the list. Isn’t that restricted >> for > members only (by default)? > > Andor > > > >> On 2020. Jul 28., at 3:15, 张铎(Duo Zhang) >> wrote: >> >> Hi Andor, >> >> The Apache-HBase account is registered by me, using the private@hbase >> mailing list, so all the PMC members can maintain the password. >> >> I generated an access token and added it to our jenkins, so we can >> use it >> to post comments back to GitHub. >> >> I think you could do the same to register an Apache-ZooKeeper >> account? Or >> if you want to use the hadoop-yetus account, you'd better ask the >> hadoop >> PMC members or Gavin to add the token to jenkins so you can use it. >> >> Thanks. >> >> Andor Molnar 于2020年7月28日周二 上午3:56写道： >> >>> Hi Duo, >>> >>> I’m trying to create a similar job for Apache ZooKeeper, but > unfortunately >>> haven’t got too much help on the Apache builds@ list so far, so I’m >>> rather asking you if you don’t mind. >>> >>> First, how have you set up the Hbase Github account that you use in >> this >>> job to access the repo? >>> >>> Thanks, >>> Andor >>> >>> >>> On 2020. Jul 27., at 2:22, 张铎(Duo Zhang) > wrote: The pre commit job has been migrated to c-hadoop.a.o. I have disabled periodical scan for the old job on builds.a.o, as we >>> still need to view the pre commit result on it do not delete for now. Will >>> delete it later, maybe after several weeks. The new job is here >> https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/ Thanks. 张铎(Duo Zhang) 于2020年7月25日周六 下午9:44写道： > > >>> > >> https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/5/console > > > We successfully finished a nightly build. > > But seems the jiraComment did not work. I haven't seen the comment > on HBASE-24757... > > 张铎(Duo Zhang) 于2020年7月25日周六 下午4:51写道： > >> After installing two new jenkins plugins, the pre commit job seems > fine >> now. >> >> The last failure is because of a timeout, I assume the problem is > that >>> we >> do not have enough executors so all the jobs are executed > sequentially. >> >> Maybe we could move the pre commit job to the new env first? The >>> nightly >> job and flaky job require more resources, and we need the output >> of >>> these >> jenkins jobs(the flaky test list). >> >> Thanks. >> >> >> >> 张铎(Duo Zhang) 于2020年7月24日周五 下午4:36写道： >> >>> The problem seems because of this: >>> >>> https://issues.jenkins-ci.org/browse/JENKINS-48556 >>> >>> I triggered the job again, it passed the timestamps call, and >> will >>> keep >>> an eye on it. >>> >>> 张铎(Duo Zhang) 于2020年7月21日周二 上午11:18写道： >>> On the sponsors, we could have a try. The problem here is the process of the donation? IIRC there is a >>> thread on the infra mailing list about how to donate
[jira] [Created] (HBASE-24797) Move log code out of loop
Sun Xin created HBASE-24797:
---
Summary: Move log code out of loop
Key: HBASE-24797
URL: https://issues.apache.org/jira/browse/HBASE-24797
Project: HBase
Issue Type: Bug
Components: Normalizer
Affects Versions: 3.0.0-alpha-1
Reporter: Sun Xin
Assignee: Sun Xin
Fix For: 3.0.0-alpha-1
In HMaster#normalizeRegions, maybe we shoule move the log code about
submittedPlanProcIds out of loop.
{code:java}
public boolean normalizeRegions() throws IOException {
...
final List submittedPlanProcIds = new ArrayList<>();
for (TableName table : allEnabledTables) {
...
for (NormalizationPlan plan : plans) {
long procId = plan.submit(this);
submittedPlanProcIds.add(procId);
...
}
int totalPlansSubmitted = submittedPlanProcIds.size();
if (totalPlansSubmitted > 0 && LOG.isDebugEnabled()) {
LOG.debug("Normalizer plans submitted. Total plans count: {} , procID
list: {}",
totalPlansSubmitted, submittedPlanProcIds);
}
}
...
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
