[GitHub] [hbase] Apache-HBase commented on pull request #3049: HBASE-25643 The delayed FlushRegionEntry should be removed when we ne…
Apache-HBase commented on pull request #3049: URL: https://github.com/apache/hbase/pull/3049#issuecomment-800813005 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | compile | 1m 0s | master passed | | +1 :green_heart: | shadedjars | 7m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 56s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | shadedjars | 7m 17s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 215m 20s | hbase-server in the patch passed. | | | | 244m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3049/11/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3049 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 782ffd1a512e 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3049/11/testReport/ | | Max. process+thread count | 3301 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3049/11/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3049: HBASE-25643 The delayed FlushRegionEntry should be removed when we ne…
Apache-HBase commented on pull request #3049: URL: https://github.com/apache/hbase/pull/3049#issuecomment-800811587 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 45s | master passed | | +1 :green_heart: | compile | 1m 13s | master passed | | +1 :green_heart: | shadedjars | 7m 30s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 39s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | shadedjars | 7m 30s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 209m 13s | hbase-server in the patch passed. | | | | 240m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3049/11/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3049 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ef91e135a05e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | Default Java | AdoptOpenJDK-11.0.10+9 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3049/11/testReport/ | | Max. process+thread count | 3324 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3049/11/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25671) Backport HBASE-25608 to branch-2
Shinya Yoshida created HBASE-25671: -- Summary: Backport HBASE-25608 to branch-2 Key: HBASE-25671 URL: https://issues.apache.org/jira/browse/HBASE-25671 Project: HBase Issue Type: Sub-task Reporter: Shinya Yoshida Assignee: Shinya Yoshida -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25672) Backport HBASE-25608 to branch-1
Shinya Yoshida created HBASE-25672: -- Summary: Backport HBASE-25608 to branch-1 Key: HBASE-25672 URL: https://issues.apache.org/jira/browse/HBASE-25672 Project: HBase Issue Type: Sub-task Reporter: Shinya Yoshida -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25672) Backport HBASE-25608 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-25672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinya Yoshida reassigned HBASE-25672: -- Assignee: Shinya Yoshida > Backport HBASE-25608 to branch-1 > > > Key: HBASE-25672 > URL: https://issues.apache.org/jira/browse/HBASE-25672 > Project: HBase > Issue Type: Sub-task >Reporter: Shinya Yoshida >Assignee: Shinya Yoshida >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25608) Support HFileOutputFormat locality sensitive even destination cluster is different from source cluster
[
https://issues.apache.org/jira/browse/HBASE-25608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303105#comment-17303105
]
Shinya Yoshida commented on HBASE-25608:
Thank you. I'll open subtask to backport to branch-2 and branch-1(y)
> Support HFileOutputFormat locality sensitive even destination cluster is
> different from source cluster
> --
>
> Key: HBASE-25608
> URL: https://issues.apache.org/jira/browse/HBASE-25608
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.1, 1.8.0
>Reporter: Shinya Yoshida
>Assignee: Shinya Yoshida
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Sometimes, we want to perform MR job which is source cluster and destination
> cluster is different like following for data migration, batch job and so on.
>
> {code:java}
> Configuration conf =
> HBaseConfiguration.createClusterConf(HBaseConfiguration.create(),
> sourceClusterKey);
> final Job job = Job.getInstance(conf, jobName);
> // ...
> FileOutputFormat.setOutputPath(job, new Path(outputPath));
> Scan scan = createScanner();
> TableMapReduceUtil.initTableMapperJob(
> sourceTableName, scan,
> Mapper.class,
> ImmutableBytesWritable.class, Put.class, job);
> try (Connection con =
> ConnectionFactory.createConnection(destinationClusterKey);
> Table table = con.getTable(destinationTableName);
> RegionLocator regionLocator =
> con.getRegionLocator(destinationTableName)) {
> HFileOutputFormat2.configureIncrementalLoad(job, table,
> regionLocator);
> }
> return job.waitForCompletion(true) ? 0 : 1;
> {code}
> HFileOutputFormat2 doesn't create locality-sensitive hfiles.
> We got following exception
> {code:java}
> 2021-02-24 19:55:48,298 WARN [main]
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2: there's something wrong
> when locating rowkey:
> org.apache.hadoop.hbase.TableNotFoundException: Table 'table' was not found,
> got: .
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1302)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:957)
> at
> org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:74)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:216)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:167)
> at
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
> at
> org.apache.hadoop.hbase.mapreduce.PutSortReducer.reduce(PutSortReducer.java:78)
> at
> org.apache.hadoop.hbase.mapreduce.PutSortReducer.reduce(PutSortReducer.java:43)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}
> Because it creates connection using task configuration which is configured
> for source cluster.
> Thus, it tried to connect to the source cluster and get locations for the
> table that should exist in the destination.
> {code:java}
> InetSocketAddress[] favoredNodes = null;
> if (conf.getBoolean(LOCALITY_SENSITIVE_CONF_KEY,
> DEFAULT_LOCALITY_SENSITIVE)) {
> HRegionLocation loc = null;
> String tableName = Bytes.toString(tableNameBytes);
> if
[jira] [Updated] (HBASE-25608) Support HFileOutputFormat locality sensitive even destination cluster is different from source cluster
[
https://issues.apache.org/jira/browse/HBASE-25608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shinya Yoshida updated HBASE-25608:
---
Release Note:
Added configurations to specify the ZK cluster key for remote cluster in
HFileOutputFormat2.
Default, input and output are to the cluster specified in Job configuration.
Use HFileOutputformat2#configureRemoteCluster to have output go to a remote
cluster.
HFileOutputFormat2#configureIncrementalLoad(Job, Table, RegionLocator)
configure them using Table's configuration.
You can also configure them by calling
HFileOutputFormat2#configureRemoteCluster explicitly.
> Support HFileOutputFormat locality sensitive even destination cluster is
> different from source cluster
> --
>
> Key: HBASE-25608
> URL: https://issues.apache.org/jira/browse/HBASE-25608
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.1, 1.8.0
>Reporter: Shinya Yoshida
>Assignee: Shinya Yoshida
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Sometimes, we want to perform MR job which is source cluster and destination
> cluster is different like following for data migration, batch job and so on.
>
> {code:java}
> Configuration conf =
> HBaseConfiguration.createClusterConf(HBaseConfiguration.create(),
> sourceClusterKey);
> final Job job = Job.getInstance(conf, jobName);
> // ...
> FileOutputFormat.setOutputPath(job, new Path(outputPath));
> Scan scan = createScanner();
> TableMapReduceUtil.initTableMapperJob(
> sourceTableName, scan,
> Mapper.class,
> ImmutableBytesWritable.class, Put.class, job);
> try (Connection con =
> ConnectionFactory.createConnection(destinationClusterKey);
> Table table = con.getTable(destinationTableName);
> RegionLocator regionLocator =
> con.getRegionLocator(destinationTableName)) {
> HFileOutputFormat2.configureIncrementalLoad(job, table,
> regionLocator);
> }
> return job.waitForCompletion(true) ? 0 : 1;
> {code}
> HFileOutputFormat2 doesn't create locality-sensitive hfiles.
> We got following exception
> {code:java}
> 2021-02-24 19:55:48,298 WARN [main]
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2: there's something wrong
> when locating rowkey:
> org.apache.hadoop.hbase.TableNotFoundException: Table 'table' was not found,
> got: .
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1302)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:957)
> at
> org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:74)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:216)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:167)
> at
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
> at
> org.apache.hadoop.hbase.mapreduce.PutSortReducer.reduce(PutSortReducer.java:78)
> at
> org.apache.hadoop.hbase.mapreduce.PutSortReducer.reduce(PutSortReducer.java:43)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}
> Because it creates connection using task configuration which is configured
> for source cluster.
> Thus, it tried to connect to the
[GitHub] [hbase] bitterfox commented on a change in pull request #2988: HBASE-25608 Support HFileOutputFormat locality sensitive even destination cluster is different from source cluster
bitterfox commented on a change in pull request #2988:
URL: https://github.com/apache/hbase/pull/2988#discussion_r595719496
##
File path:
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##
@@ -657,6 +683,49 @@ public static void configureIncrementalLoadMap(Job job,
TableDescriptor tableDes
LOG.info("Incremental table " + tableDescriptor.getTableName() + " output
configured.");
}
+ /**
+ * Configure HBase cluster key to load region location for
locality-sensitive it's enabled.
+ * It's not necessary to call this method explicitly when the cluster key
for HBase cluster to be
+ * used to load region location is configured in the job configuration.
+ * Call this method when another HBase cluster key is configured in the job
configuration.
+ * For example, you should call when you load data from HBase cluster A using
+ * {@link TableInputFormat} and generate hfiles for HBase cluster B.
+ * Otherwise, HFileOutputFormat2 fetch location from cluster A and
locality-sensitive won't
+ * working correctly.
+ * {@link #configureIncrementalLoad(Job, Table, RegionLocator)} calls this
method using
+ * {@link Table#getConfiguration} as clusterConf.
+ * See HBASE-25608.
+ *
+ * @param job which has configuration to be updated
+ * @param clusterConf which contains cluster key of the HBase cluster to be
locality-sensitive
+ *
+ * @see #configureIncrementalLoad(Job, Table, RegionLocator)
+ * @see #LOCALITY_SENSITIVE_CONF_KEY
+ * @see #LOCALITY_SENSITIVE_ZOOKEEPER_QUORUM_CONF_KEY
+ * @see #LOCALITY_SENSITIVE_ZOOKEEPER_CLIENT_PORT_CONF_KEY
+ * @see #LOCALITY_SENSITIVE_ZOOKEEPER_ZNODE_PARENT_CONF_KEY
+ */
+ public static void configureLocalitySensitiveCluster(Job job, Configuration
clusterConf) {
Review comment:
Thank you! I'll fix and fill release note in the issue
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25665) Disable reverse DNS lookup for SASL Kerberos client connection
[
https://issues.apache.org/jira/browse/HBASE-25665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303092#comment-17303092
]
Michael Stack commented on HBASE-25665:
---
bq. I'd like to backport this option to branch-1, thus I'll create a subtask
and another PR soon.
(y)
> Disable reverse DNS lookup for SASL Kerberos client connection
> --
>
> Key: HBASE-25665
> URL: https://issues.apache.org/jira/browse/HBASE-25665
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 3.0.0-alpha-1, 1.4.13, 2.4.1
>Reporter: Shinya Yoshida
>Assignee: Shinya Yoshida
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> In some unusual network environment that forward DNS lookup is supported, but
> revers isn't,
> we can configure the HBase cluster by deploying/etc/hosts which support
> reverse lookup for all nodes in the cluster or
> hbase.unsafe.regionserver.hostname.disable.master.reversedns=true which is
> introduced in HBASE-18226(See also HBASE-12954, HBASE-24667).
> Our network environment is also unusual and doesn't provide a reverse lookup.
> Thus, we configure the HBase cluster by deploying /etc/hosts.
> Assume our DNS setup is
> {code:java}
> master1.example.com A 12.34.56.1
> master2.example.com A 12.34.56.2
> master3.example.com A 12.34.56.3
> regionserver1.example.com A 12.34.56.4
> regionserver2.example.com A 12.34.56.5
> regionserver3.example.com A 12.34.56.6
> {code}
> We deploy the following /etc/hosts for the HBase cluster nodes by our
> deployment system.
> {code:java}
> 12.34.56.1 master1.example.com master1
> 12.34.56.2 master2.example.com master2
> 12.34.56.3 master3.example.com master3
> 12.34.56.4 regionserver1.example.com regionserver1
> 12.34.56.5 regionserver2.example.com regionserver2
> 12.34.56.6 regionserver3.example.com regionserver3
> {code}
> (We don't use
> hbase.unsafe.regionserver.hostname.disable.master.reversedns=true for now)
> So all nodes in the cluster have stable reverse lookup for the IPs in the
> cluster, and the HBase cluster deployed in this way is quite stable and we
> can expand the cluster easily without any modification on the client-side.
> Now we need to introduce Kerberos SASL secured cluster for security reasons.
> We tried to construct in the same way as is, i.e. deploy /etc/hosts for the
> HBase cluster nodes.
> However, this won't work well because the HBase client does a reverse lookup
> to get principal for Kerberos.
> (hbase.unsafe.regionserver.hostname.disable.master.reversedns=true won't work
> as well as)
> Thus we need to deploy /etc/hosts to all application servers, which contains
> all nodes of the HBase cluster to be connected.
> This is quite terrible for our cluster operation and application server setup.
> We, the HBase cluster manager, need to take care of application server setup
> and deployment.
> We must provide all master and region server lists.
> It's much more complicated when applications access multiple HBase clusters...
> We, the HBase cluster manager, cannot expand the cluster unless the latest
> /etc/hosts are deployed to all application servers.
> If we expand the cluster before deployment, the application is unable to
> connect and got an error
> Assume their own Kerberos principal is their FQDN i.e. master1.example.com
> for example, and the cluster is aware of their FQDN.
> So all clients can connect cluster nodes using the FQDN for Kerberos
> principal.
> Could we provide an advanced unsafe option to disable DNS reverse lookup for
> clients using Kerberos SASL like
> hbase.unsafe.regionserver.hostname.disable.master.reversedns and other config?
> Let's say `hbase.unsafe.client.kerberos.hostname.disable.reversedns` and if
> this is true, client uses InetAddress.getHostname() for Kerberos principal
> instead of InetAddress.getCanonicalHostName().
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25608) Support HFileOutputFormat locality sensitive even destination cluster is different from source cluster
[
https://issues.apache.org/jira/browse/HBASE-25608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-25608:
--
Fix Version/s: 3.0.0-alpha-1
> Support HFileOutputFormat locality sensitive even destination cluster is
> different from source cluster
> --
>
> Key: HBASE-25608
> URL: https://issues.apache.org/jira/browse/HBASE-25608
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.1, 1.8.0
>Reporter: Shinya Yoshida
>Assignee: Shinya Yoshida
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Sometimes, we want to perform MR job which is source cluster and destination
> cluster is different like following for data migration, batch job and so on.
>
> {code:java}
> Configuration conf =
> HBaseConfiguration.createClusterConf(HBaseConfiguration.create(),
> sourceClusterKey);
> final Job job = Job.getInstance(conf, jobName);
> // ...
> FileOutputFormat.setOutputPath(job, new Path(outputPath));
> Scan scan = createScanner();
> TableMapReduceUtil.initTableMapperJob(
> sourceTableName, scan,
> Mapper.class,
> ImmutableBytesWritable.class, Put.class, job);
> try (Connection con =
> ConnectionFactory.createConnection(destinationClusterKey);
> Table table = con.getTable(destinationTableName);
> RegionLocator regionLocator =
> con.getRegionLocator(destinationTableName)) {
> HFileOutputFormat2.configureIncrementalLoad(job, table,
> regionLocator);
> }
> return job.waitForCompletion(true) ? 0 : 1;
> {code}
> HFileOutputFormat2 doesn't create locality-sensitive hfiles.
> We got following exception
> {code:java}
> 2021-02-24 19:55:48,298 WARN [main]
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2: there's something wrong
> when locating rowkey:
> org.apache.hadoop.hbase.TableNotFoundException: Table 'table' was not found,
> got: .
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1302)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:957)
> at
> org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:74)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:216)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:167)
> at
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
> at
> org.apache.hadoop.hbase.mapreduce.PutSortReducer.reduce(PutSortReducer.java:78)
> at
> org.apache.hadoop.hbase.mapreduce.PutSortReducer.reduce(PutSortReducer.java:43)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}
> Because it creates connection using task configuration which is configured
> for source cluster.
> Thus, it tried to connect to the source cluster and get locations for the
> table that should exist in the destination.
> {code:java}
> InetSocketAddress[] favoredNodes = null;
> if (conf.getBoolean(LOCALITY_SENSITIVE_CONF_KEY,
> DEFAULT_LOCALITY_SENSITIVE)) {
> HRegionLocation loc = null;
> String tableName = Bytes.toString(tableNameBytes);
> if (tableName != null) {
> try (Connection connection =
>
[jira] [Resolved] (HBASE-25608) Support HFileOutputFormat locality sensitive even destination cluster is different from source cluster
[
https://issues.apache.org/jira/browse/HBASE-25608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-25608.
---
Hadoop Flags: Reviewed
Resolution: Fixed
Merged to master. Thanks for the nice feature [~lineyshinya]. I tried to
backport it to branch-2 but it would not go back w/o complaint. Please open a
subtask if you want to backport. Release note would be nice too for this nice
addtion. Thank you.
> Support HFileOutputFormat locality sensitive even destination cluster is
> different from source cluster
> --
>
> Key: HBASE-25608
> URL: https://issues.apache.org/jira/browse/HBASE-25608
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 1.7.0, 2.4.1, 1.8.0
>Reporter: Shinya Yoshida
>Assignee: Shinya Yoshida
>Priority: Major
>
> Sometimes, we want to perform MR job which is source cluster and destination
> cluster is different like following for data migration, batch job and so on.
>
> {code:java}
> Configuration conf =
> HBaseConfiguration.createClusterConf(HBaseConfiguration.create(),
> sourceClusterKey);
> final Job job = Job.getInstance(conf, jobName);
> // ...
> FileOutputFormat.setOutputPath(job, new Path(outputPath));
> Scan scan = createScanner();
> TableMapReduceUtil.initTableMapperJob(
> sourceTableName, scan,
> Mapper.class,
> ImmutableBytesWritable.class, Put.class, job);
> try (Connection con =
> ConnectionFactory.createConnection(destinationClusterKey);
> Table table = con.getTable(destinationTableName);
> RegionLocator regionLocator =
> con.getRegionLocator(destinationTableName)) {
> HFileOutputFormat2.configureIncrementalLoad(job, table,
> regionLocator);
> }
> return job.waitForCompletion(true) ? 0 : 1;
> {code}
> HFileOutputFormat2 doesn't create locality-sensitive hfiles.
> We got following exception
> {code:java}
> 2021-02-24 19:55:48,298 WARN [main]
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2: there's something wrong
> when locating rowkey:
> org.apache.hadoop.hbase.TableNotFoundException: Table 'table' was not found,
> got: .
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1302)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:957)
> at
> org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:74)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:216)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:167)
> at
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
> at
> org.apache.hadoop.hbase.mapreduce.PutSortReducer.reduce(PutSortReducer.java:78)
> at
> org.apache.hadoop.hbase.mapreduce.PutSortReducer.reduce(PutSortReducer.java:43)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}
> Because it creates connection using task configuration which is configured
> for source cluster.
> Thus, it tried to connect to the source cluster and get locations for the
> table that should exist in the destination.
> {code:java}
> InetSocketAddress[] favoredNodes = null;
> if (conf.getBoolean(LOCALITY_SENSITIVE_CONF_KEY,
>
[GitHub] [hbase] saintstack merged pull request #2988: HBASE-25608 Support HFileOutputFormat locality sensitive even destination cluster is different from source cluster
saintstack merged pull request #2988: URL: https://github.com/apache/hbase/pull/2988 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2988: HBASE-25608 Support HFileOutputFormat locality sensitive even destination cluster is different from source cluster
saintstack commented on a change in pull request #2988:
URL: https://github.com/apache/hbase/pull/2988#discussion_r595709760
##
File path:
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##
@@ -657,6 +683,49 @@ public static void configureIncrementalLoadMap(Job job,
TableDescriptor tableDes
LOG.info("Incremental table " + tableDescriptor.getTableName() + " output
configured.");
}
+ /**
+ * Configure HBase cluster key to load region location for
locality-sensitive it's enabled.
+ * It's not necessary to call this method explicitly when the cluster key
for HBase cluster to be
+ * used to load region location is configured in the job configuration.
+ * Call this method when another HBase cluster key is configured in the job
configuration.
+ * For example, you should call when you load data from HBase cluster A using
+ * {@link TableInputFormat} and generate hfiles for HBase cluster B.
+ * Otherwise, HFileOutputFormat2 fetch location from cluster A and
locality-sensitive won't
+ * working correctly.
+ * {@link #configureIncrementalLoad(Job, Table, RegionLocator)} calls this
method using
+ * {@link Table#getConfiguration} as clusterConf.
+ * See HBASE-25608.
+ *
+ * @param job which has configuration to be updated
+ * @param clusterConf which contains cluster key of the HBase cluster to be
locality-sensitive
+ *
+ * @see #configureIncrementalLoad(Job, Table, RegionLocator)
+ * @see #LOCALITY_SENSITIVE_CONF_KEY
+ * @see #LOCALITY_SENSITIVE_ZOOKEEPER_QUORUM_CONF_KEY
+ * @see #LOCALITY_SENSITIVE_ZOOKEEPER_CLIENT_PORT_CONF_KEY
+ * @see #LOCALITY_SENSITIVE_ZOOKEEPER_ZNODE_PARENT_CONF_KEY
+ */
+ public static void configureLocalitySensitiveCluster(Job job, Configuration
clusterConf) {
Review comment:
s/for the remote/for remote/
Remove 'These generation.'
s/Previouslydifferent./Default, input and output are to the cluster
specified in Job configuration.
Use HFileOutputformat2#configureRemoteCluster to have output go to a remote
cluster.
Something like that?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (HBASE-25653) Improve normalizer logging
[ https://issues.apache.org/jira/browse/HBASE-25653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-25653. --- Fix Version/s: 3.0.0-alpha-1 Hadoop Flags: Reviewed Resolution: Fixed Merged to Master. I tried to cherry-pick it back but failed to pick cleanly. Please make subtasks to backport beyond master if you want it to go back more [~divyesh]. Thanks for the PR. > Improve normalizer logging > -- > > Key: HBASE-25653 > URL: https://issues.apache.org/jira/browse/HBASE-25653 > Project: HBase > Issue Type: Improvement > Components: master, Normalizer >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Divyesh Chandra >Priority: Major > Fix For: 3.0.0-alpha-1 > > > Normalizer logs progress and includes details regarding region count and > target size. The size values are logged without units, and floating point > numbers are logged without a specified precision, which can result in > scientific notation. Clean up the formatting of these log messages to make > them easily readable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #3046: HBASE-25653 Add units and round off region size to 2 digits after decimal
saintstack merged pull request #3046: URL: https://github.com/apache/hbase/pull/3046 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25670) Backport HBASE-25665 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-25670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinya Yoshida updated HBASE-25670: --- Description: Backport [https://github.com/apache/hbase/commit/ebb0adf50009fc133af0cfb0bdce4dfbb81d4fbf] for https://issues.apache.org/jira/browse/HBASE-25665 to branch-1 > Backport HBASE-25665 to branch-1 > > > Key: HBASE-25670 > URL: https://issues.apache.org/jira/browse/HBASE-25670 > Project: HBase > Issue Type: Sub-task >Reporter: Shinya Yoshida >Assignee: Shinya Yoshida >Priority: Major > > Backport > [https://github.com/apache/hbase/commit/ebb0adf50009fc133af0cfb0bdce4dfbb81d4fbf] > for https://issues.apache.org/jira/browse/HBASE-25665 to branch-1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25670) Backport HBASE-25665 to branch-1
Shinya Yoshida created HBASE-25670: -- Summary: Backport HBASE-25665 to branch-1 Key: HBASE-25670 URL: https://issues.apache.org/jira/browse/HBASE-25670 Project: HBase Issue Type: Sub-task Reporter: Shinya Yoshida Assignee: Shinya Yoshida -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25665) Disable reverse DNS lookup for SASL Kerberos client connection
[
https://issues.apache.org/jira/browse/HBASE-25665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303083#comment-17303083
]
Shinya Yoshida commented on HBASE-25665:
Hi, Thank you for reviewing and merging the change!
I added the release note for this ticket.
I'd like to backport this option to branch-1, thus I'll create a subtask and
another PR soon.
> Disable reverse DNS lookup for SASL Kerberos client connection
> --
>
> Key: HBASE-25665
> URL: https://issues.apache.org/jira/browse/HBASE-25665
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 3.0.0-alpha-1, 1.4.13, 2.4.1
>Reporter: Shinya Yoshida
>Assignee: Shinya Yoshida
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> In some unusual network environment that forward DNS lookup is supported, but
> revers isn't,
> we can configure the HBase cluster by deploying/etc/hosts which support
> reverse lookup for all nodes in the cluster or
> hbase.unsafe.regionserver.hostname.disable.master.reversedns=true which is
> introduced in HBASE-18226(See also HBASE-12954, HBASE-24667).
> Our network environment is also unusual and doesn't provide a reverse lookup.
> Thus, we configure the HBase cluster by deploying /etc/hosts.
> Assume our DNS setup is
> {code:java}
> master1.example.com A 12.34.56.1
> master2.example.com A 12.34.56.2
> master3.example.com A 12.34.56.3
> regionserver1.example.com A 12.34.56.4
> regionserver2.example.com A 12.34.56.5
> regionserver3.example.com A 12.34.56.6
> {code}
> We deploy the following /etc/hosts for the HBase cluster nodes by our
> deployment system.
> {code:java}
> 12.34.56.1 master1.example.com master1
> 12.34.56.2 master2.example.com master2
> 12.34.56.3 master3.example.com master3
> 12.34.56.4 regionserver1.example.com regionserver1
> 12.34.56.5 regionserver2.example.com regionserver2
> 12.34.56.6 regionserver3.example.com regionserver3
> {code}
> (We don't use
> hbase.unsafe.regionserver.hostname.disable.master.reversedns=true for now)
> So all nodes in the cluster have stable reverse lookup for the IPs in the
> cluster, and the HBase cluster deployed in this way is quite stable and we
> can expand the cluster easily without any modification on the client-side.
> Now we need to introduce Kerberos SASL secured cluster for security reasons.
> We tried to construct in the same way as is, i.e. deploy /etc/hosts for the
> HBase cluster nodes.
> However, this won't work well because the HBase client does a reverse lookup
> to get principal for Kerberos.
> (hbase.unsafe.regionserver.hostname.disable.master.reversedns=true won't work
> as well as)
> Thus we need to deploy /etc/hosts to all application servers, which contains
> all nodes of the HBase cluster to be connected.
> This is quite terrible for our cluster operation and application server setup.
> We, the HBase cluster manager, need to take care of application server setup
> and deployment.
> We must provide all master and region server lists.
> It's much more complicated when applications access multiple HBase clusters...
> We, the HBase cluster manager, cannot expand the cluster unless the latest
> /etc/hosts are deployed to all application servers.
> If we expand the cluster before deployment, the application is unable to
> connect and got an error
> Assume their own Kerberos principal is their FQDN i.e. master1.example.com
> for example, and the cluster is aware of their FQDN.
> So all clients can connect cluster nodes using the FQDN for Kerberos
> principal.
> Could we provide an advanced unsafe option to disable DNS reverse lookup for
> clients using Kerberos SASL like
> hbase.unsafe.regionserver.hostname.disable.master.reversedns and other config?
> Let's say `hbase.unsafe.client.kerberos.hostname.disable.reversedns` and if
> this is true, client uses InetAddress.getHostname() for Kerberos principal
> instead of InetAddress.getCanonicalHostName().
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25665) Disable reverse DNS lookup for SASL Kerberos client connection
[
https://issues.apache.org/jira/browse/HBASE-25665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shinya Yoshida updated HBASE-25665:
---
Release Note:
New client side configuration
`hbase.unsafe.client.kerberos.hostname.disable.reversedns` is added.
This configuration is advanced for experts and you shouldn't specify unless you
really what is this configuration and doing.
HBase secure client using SASL Kerberos performs DNS reverse lookup to get
hostname for server principal using InetAddress.getCanonicalHostName by default
(false for this configuration).
If you set true for this configuration, HBase client doen't perform DNS reverse
lookup for server principal and use InetAddress.getHostName which is sent by
HBase cluster instead.
This helps your client application deploy under unusual network environment
which DNS doesn't provide reverse lookup.
Check the description of the JIRA ticket, HBASE-25665 carefully and check that
this configuration fits your environment and deployment actually before enable
this configuration.
> Disable reverse DNS lookup for SASL Kerberos client connection
> --
>
> Key: HBASE-25665
> URL: https://issues.apache.org/jira/browse/HBASE-25665
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 3.0.0-alpha-1, 1.4.13, 2.4.1
>Reporter: Shinya Yoshida
>Assignee: Shinya Yoshida
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> In some unusual network environment that forward DNS lookup is supported, but
> revers isn't,
> we can configure the HBase cluster by deploying/etc/hosts which support
> reverse lookup for all nodes in the cluster or
> hbase.unsafe.regionserver.hostname.disable.master.reversedns=true which is
> introduced in HBASE-18226(See also HBASE-12954, HBASE-24667).
> Our network environment is also unusual and doesn't provide a reverse lookup.
> Thus, we configure the HBase cluster by deploying /etc/hosts.
> Assume our DNS setup is
> {code:java}
> master1.example.com A 12.34.56.1
> master2.example.com A 12.34.56.2
> master3.example.com A 12.34.56.3
> regionserver1.example.com A 12.34.56.4
> regionserver2.example.com A 12.34.56.5
> regionserver3.example.com A 12.34.56.6
> {code}
> We deploy the following /etc/hosts for the HBase cluster nodes by our
> deployment system.
> {code:java}
> 12.34.56.1 master1.example.com master1
> 12.34.56.2 master2.example.com master2
> 12.34.56.3 master3.example.com master3
> 12.34.56.4 regionserver1.example.com regionserver1
> 12.34.56.5 regionserver2.example.com regionserver2
> 12.34.56.6 regionserver3.example.com regionserver3
> {code}
> (We don't use
> hbase.unsafe.regionserver.hostname.disable.master.reversedns=true for now)
> So all nodes in the cluster have stable reverse lookup for the IPs in the
> cluster, and the HBase cluster deployed in this way is quite stable and we
> can expand the cluster easily without any modification on the client-side.
> Now we need to introduce Kerberos SASL secured cluster for security reasons.
> We tried to construct in the same way as is, i.e. deploy /etc/hosts for the
> HBase cluster nodes.
> However, this won't work well because the HBase client does a reverse lookup
> to get principal for Kerberos.
> (hbase.unsafe.regionserver.hostname.disable.master.reversedns=true won't work
> as well as)
> Thus we need to deploy /etc/hosts to all application servers, which contains
> all nodes of the HBase cluster to be connected.
> This is quite terrible for our cluster operation and application server setup.
> We, the HBase cluster manager, need to take care of application server setup
> and deployment.
> We must provide all master and region server lists.
> It's much more complicated when applications access multiple HBase clusters...
> We, the HBase cluster manager, cannot expand the cluster unless the latest
> /etc/hosts are deployed to all application servers.
> If we expand the cluster before deployment, the application is unable to
> connect and got an error
> Assume their own Kerberos principal is their FQDN i.e. master1.example.com
> for example, and the cluster is aware of their FQDN.
> So all clients can connect cluster nodes using the FQDN for Kerberos
> principal.
> Could we provide an advanced unsafe option to disable DNS reverse lookup for
> clients using Kerberos SASL like
> hbase.unsafe.regionserver.hostname.disable.master.reversedns and other config?
> Let's say `hbase.unsafe.client.kerberos.hostname.disable.reversedns` and if
> this is true, client uses InetAddress.getHostname() for Kerberos principal
> instead of InetAddress.getCanonicalHostName().
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Apache-HBase commented on pull request #3001: URL: https://github.com/apache/hbase/pull/3001#issuecomment-800778549 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 33s | master passed | | +1 :green_heart: | compile | 4m 2s | master passed | | +1 :green_heart: | checkstyle | 1m 29s | master passed | | +1 :green_heart: | spotbugs | 2m 46s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 30s | the patch passed | | +1 :green_heart: | compile | 4m 5s | the patch passed | | +1 :green_heart: | javac | 4m 5s | the patch passed | | -0 :warning: | checkstyle | 1m 5s | hbase-server: The patch generated 3 new + 1 unchanged - 1 fixed = 4 total (was 2) | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 22s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 50m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3001 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4a355c869731 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 95 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/4/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25665) Disable reverse DNS lookup for SASL Kerberos client connection
[
https://issues.apache.org/jira/browse/HBASE-25665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303078#comment-17303078
]
Michael Stack commented on HBASE-25665:
---
Changed my mind. Its a new feature so removed it from branch-2.4. Its on
branch-2 and on master. If you want it on branch-1, make a subtask and I'll
push for you. Thank you [~lineyshinya]
> Disable reverse DNS lookup for SASL Kerberos client connection
> --
>
> Key: HBASE-25665
> URL: https://issues.apache.org/jira/browse/HBASE-25665
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 3.0.0-alpha-1, 1.4.13, 2.4.1
>Reporter: Shinya Yoshida
>Assignee: Shinya Yoshida
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> In some unusual network environment that forward DNS lookup is supported, but
> revers isn't,
> we can configure the HBase cluster by deploying/etc/hosts which support
> reverse lookup for all nodes in the cluster or
> hbase.unsafe.regionserver.hostname.disable.master.reversedns=true which is
> introduced in HBASE-18226(See also HBASE-12954, HBASE-24667).
> Our network environment is also unusual and doesn't provide a reverse lookup.
> Thus, we configure the HBase cluster by deploying /etc/hosts.
> Assume our DNS setup is
> {code:java}
> master1.example.com A 12.34.56.1
> master2.example.com A 12.34.56.2
> master3.example.com A 12.34.56.3
> regionserver1.example.com A 12.34.56.4
> regionserver2.example.com A 12.34.56.5
> regionserver3.example.com A 12.34.56.6
> {code}
> We deploy the following /etc/hosts for the HBase cluster nodes by our
> deployment system.
> {code:java}
> 12.34.56.1 master1.example.com master1
> 12.34.56.2 master2.example.com master2
> 12.34.56.3 master3.example.com master3
> 12.34.56.4 regionserver1.example.com regionserver1
> 12.34.56.5 regionserver2.example.com regionserver2
> 12.34.56.6 regionserver3.example.com regionserver3
> {code}
> (We don't use
> hbase.unsafe.regionserver.hostname.disable.master.reversedns=true for now)
> So all nodes in the cluster have stable reverse lookup for the IPs in the
> cluster, and the HBase cluster deployed in this way is quite stable and we
> can expand the cluster easily without any modification on the client-side.
> Now we need to introduce Kerberos SASL secured cluster for security reasons.
> We tried to construct in the same way as is, i.e. deploy /etc/hosts for the
> HBase cluster nodes.
> However, this won't work well because the HBase client does a reverse lookup
> to get principal for Kerberos.
> (hbase.unsafe.regionserver.hostname.disable.master.reversedns=true won't work
> as well as)
> Thus we need to deploy /etc/hosts to all application servers, which contains
> all nodes of the HBase cluster to be connected.
> This is quite terrible for our cluster operation and application server setup.
> We, the HBase cluster manager, need to take care of application server setup
> and deployment.
> We must provide all master and region server lists.
> It's much more complicated when applications access multiple HBase clusters...
> We, the HBase cluster manager, cannot expand the cluster unless the latest
> /etc/hosts are deployed to all application servers.
> If we expand the cluster before deployment, the application is unable to
> connect and got an error
> Assume their own Kerberos principal is their FQDN i.e. master1.example.com
> for example, and the cluster is aware of their FQDN.
> So all clients can connect cluster nodes using the FQDN for Kerberos
> principal.
> Could we provide an advanced unsafe option to disable DNS reverse lookup for
> clients using Kerberos SASL like
> hbase.unsafe.regionserver.hostname.disable.master.reversedns and other config?
> Let's say `hbase.unsafe.client.kerberos.hostname.disable.reversedns` and if
> this is true, client uses InetAddress.getHostname() for Kerberos principal
> instead of InetAddress.getCanonicalHostName().
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25665) Disable reverse DNS lookup for SASL Kerberos client connection
[
https://issues.apache.org/jira/browse/HBASE-25665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack updated HBASE-25665:
--
Fix Version/s: (was: 2.4.3)
> Disable reverse DNS lookup for SASL Kerberos client connection
> --
>
> Key: HBASE-25665
> URL: https://issues.apache.org/jira/browse/HBASE-25665
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 3.0.0-alpha-1, 1.4.13, 2.4.1
>Reporter: Shinya Yoshida
>Assignee: Shinya Yoshida
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> In some unusual network environment that forward DNS lookup is supported, but
> revers isn't,
> we can configure the HBase cluster by deploying/etc/hosts which support
> reverse lookup for all nodes in the cluster or
> hbase.unsafe.regionserver.hostname.disable.master.reversedns=true which is
> introduced in HBASE-18226(See also HBASE-12954, HBASE-24667).
> Our network environment is also unusual and doesn't provide a reverse lookup.
> Thus, we configure the HBase cluster by deploying /etc/hosts.
> Assume our DNS setup is
> {code:java}
> master1.example.com A 12.34.56.1
> master2.example.com A 12.34.56.2
> master3.example.com A 12.34.56.3
> regionserver1.example.com A 12.34.56.4
> regionserver2.example.com A 12.34.56.5
> regionserver3.example.com A 12.34.56.6
> {code}
> We deploy the following /etc/hosts for the HBase cluster nodes by our
> deployment system.
> {code:java}
> 12.34.56.1 master1.example.com master1
> 12.34.56.2 master2.example.com master2
> 12.34.56.3 master3.example.com master3
> 12.34.56.4 regionserver1.example.com regionserver1
> 12.34.56.5 regionserver2.example.com regionserver2
> 12.34.56.6 regionserver3.example.com regionserver3
> {code}
> (We don't use
> hbase.unsafe.regionserver.hostname.disable.master.reversedns=true for now)
> So all nodes in the cluster have stable reverse lookup for the IPs in the
> cluster, and the HBase cluster deployed in this way is quite stable and we
> can expand the cluster easily without any modification on the client-side.
> Now we need to introduce Kerberos SASL secured cluster for security reasons.
> We tried to construct in the same way as is, i.e. deploy /etc/hosts for the
> HBase cluster nodes.
> However, this won't work well because the HBase client does a reverse lookup
> to get principal for Kerberos.
> (hbase.unsafe.regionserver.hostname.disable.master.reversedns=true won't work
> as well as)
> Thus we need to deploy /etc/hosts to all application servers, which contains
> all nodes of the HBase cluster to be connected.
> This is quite terrible for our cluster operation and application server setup.
> We, the HBase cluster manager, need to take care of application server setup
> and deployment.
> We must provide all master and region server lists.
> It's much more complicated when applications access multiple HBase clusters...
> We, the HBase cluster manager, cannot expand the cluster unless the latest
> /etc/hosts are deployed to all application servers.
> If we expand the cluster before deployment, the application is unable to
> connect and got an error
> Assume their own Kerberos principal is their FQDN i.e. master1.example.com
> for example, and the cluster is aware of their FQDN.
> So all clients can connect cluster nodes using the FQDN for Kerberos
> principal.
> Could we provide an advanced unsafe option to disable DNS reverse lookup for
> clients using Kerberos SASL like
> hbase.unsafe.regionserver.hostname.disable.master.reversedns and other config?
> Let's say `hbase.unsafe.client.kerberos.hostname.disable.reversedns` and if
> this is true, client uses InetAddress.getHostname() for Kerberos principal
> instead of InetAddress.getCanonicalHostName().
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-25665) Disable reverse DNS lookup for SASL Kerberos client connection
[
https://issues.apache.org/jira/browse/HBASE-25665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-25665.
---
Fix Version/s: 2.4.3
2.5.0
3.0.0-alpha-1
Hadoop Flags: Reviewed
Resolution: Fixed
Pushed to 2.4+. Thanks for PR [~lineyshinya] Please add a release note
describing your nice new addition.
> Disable reverse DNS lookup for SASL Kerberos client connection
> --
>
> Key: HBASE-25665
> URL: https://issues.apache.org/jira/browse/HBASE-25665
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 3.0.0-alpha-1, 1.4.13, 2.4.1
>Reporter: Shinya Yoshida
>Assignee: Shinya Yoshida
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> In some unusual network environment that forward DNS lookup is supported, but
> revers isn't,
> we can configure the HBase cluster by deploying/etc/hosts which support
> reverse lookup for all nodes in the cluster or
> hbase.unsafe.regionserver.hostname.disable.master.reversedns=true which is
> introduced in HBASE-18226(See also HBASE-12954, HBASE-24667).
> Our network environment is also unusual and doesn't provide a reverse lookup.
> Thus, we configure the HBase cluster by deploying /etc/hosts.
> Assume our DNS setup is
> {code:java}
> master1.example.com A 12.34.56.1
> master2.example.com A 12.34.56.2
> master3.example.com A 12.34.56.3
> regionserver1.example.com A 12.34.56.4
> regionserver2.example.com A 12.34.56.5
> regionserver3.example.com A 12.34.56.6
> {code}
> We deploy the following /etc/hosts for the HBase cluster nodes by our
> deployment system.
> {code:java}
> 12.34.56.1 master1.example.com master1
> 12.34.56.2 master2.example.com master2
> 12.34.56.3 master3.example.com master3
> 12.34.56.4 regionserver1.example.com regionserver1
> 12.34.56.5 regionserver2.example.com regionserver2
> 12.34.56.6 regionserver3.example.com regionserver3
> {code}
> (We don't use
> hbase.unsafe.regionserver.hostname.disable.master.reversedns=true for now)
> So all nodes in the cluster have stable reverse lookup for the IPs in the
> cluster, and the HBase cluster deployed in this way is quite stable and we
> can expand the cluster easily without any modification on the client-side.
> Now we need to introduce Kerberos SASL secured cluster for security reasons.
> We tried to construct in the same way as is, i.e. deploy /etc/hosts for the
> HBase cluster nodes.
> However, this won't work well because the HBase client does a reverse lookup
> to get principal for Kerberos.
> (hbase.unsafe.regionserver.hostname.disable.master.reversedns=true won't work
> as well as)
> Thus we need to deploy /etc/hosts to all application servers, which contains
> all nodes of the HBase cluster to be connected.
> This is quite terrible for our cluster operation and application server setup.
> We, the HBase cluster manager, need to take care of application server setup
> and deployment.
> We must provide all master and region server lists.
> It's much more complicated when applications access multiple HBase clusters...
> We, the HBase cluster manager, cannot expand the cluster unless the latest
> /etc/hosts are deployed to all application servers.
> If we expand the cluster before deployment, the application is unable to
> connect and got an error
> Assume their own Kerberos principal is their FQDN i.e. master1.example.com
> for example, and the cluster is aware of their FQDN.
> So all clients can connect cluster nodes using the FQDN for Kerberos
> principal.
> Could we provide an advanced unsafe option to disable DNS reverse lookup for
> clients using Kerberos SASL like
> hbase.unsafe.regionserver.hostname.disable.master.reversedns and other config?
> Let's say `hbase.unsafe.client.kerberos.hostname.disable.reversedns` and if
> this is true, client uses InetAddress.getHostname() for Kerberos principal
> instead of InetAddress.getCanonicalHostName().
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #3051: HBASE-25665 Option to use hostname instead of canonical hostname for secure HBase cluster connection
saintstack merged pull request #3051: URL: https://github.com/apache/hbase/pull/3051 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #3052: HBASE-25639 meta replica state is not respected during active master …
saintstack commented on a change in pull request #3052:
URL: https://github.com/apache/hbase/pull/3052#discussion_r595697207
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -1018,7 +1018,7 @@ private void
finishActiveMasterInitialization(MonitoredTask status)
RegionState rs = this.assignmentManager.getRegionStates().
getRegionState(RegionInfoBuilder.FIRST_META_REGIONINFO);
LOG.info("hbase:meta {}", rs);
-if (rs != null && rs.isOffline()) {
+if ((rs == null) || (rs != null && rs.isOffline())) {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #3052: HBASE-25639 meta replica state is not respected during active master …
saintstack commented on a change in pull request #3052:
URL: https://github.com/apache/hbase/pull/3052#discussion_r595697030
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java
##
@@ -61,25 +60,25 @@ void assignMetaReplicas()
throw new IllegalStateException("hbase:meta must be initialized first
before we can " +
"assign out its replicas");
}
-ServerName metaServername =
MetaTableLocator.getMetaRegionLocation(this.master.getZooKeeper());
+
for (int i = 1; i < numReplicas; i++) {
// Get current meta state for replica from zk.
- RegionState metaState =
MetaTableLocator.getMetaRegionState(master.getZooKeeper(), i);
RegionInfo hri = RegionReplicaUtil.getRegionInfoForReplica(
- RegionInfoBuilder.FIRST_META_REGIONINFO, i);
- LOG.debug(hri.getRegionNameAsString() + " replica region state from
zookeeper=" + metaState);
- if (metaServername.equals(metaState.getServerName())) {
-metaState = null;
-LOG.info(hri.getRegionNameAsString() +
- " old location is same as current hbase:meta location; setting
location as null...");
- }
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (HBASE-25669) Typo in docs vs code: hbase.mob.[file.]compaction.chore.period
[ https://issues.apache.org/jira/browse/HBASE-25669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-25669. --- Fix Version/s: 3.0.0-alpha-1 Hadoop Flags: Reviewed Resolution: Fixed Merged to master. Current docs are kept on master. If a 2.2 release wants to update doc., it'll pick special from master. Thanks [~nkalmar] > Typo in docs vs code: hbase.mob.[file.]compaction.chore.period > -- > > Key: HBASE-25669 > URL: https://issues.apache.org/jira/browse/HBASE-25669 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1 >Reporter: Norbert Kalmár >Assignee: Norbert Kalmár >Priority: Major > Fix For: 3.0.0-alpha-1 > > > I might be missing something here, but after trying to disable mob compaction > chore, and failed at it by setting hbase.mob.file.compaction.chore.period to > 0, I realized docs and the code differ. > Code: > https://github.com/apache/hbase/search?q=hbase.mob.compaction.chore.period > Docs: > https://github.com/apache/hbase/search?q=hbase.mob.file.compaction.chore.period > Looks like only a portion of the docs, still confusing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #3056: HBASE-25669 Fix typo of hbase.mob.compaction.chore.period in the docs
saintstack merged pull request #3056: URL: https://github.com/apache/hbase/pull/3056 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3049: HBASE-25643 The delayed FlushRegionEntry should be removed when we ne…
Apache-HBase commented on pull request #3049: URL: https://github.com/apache/hbase/pull/3049#issuecomment-800745141 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 52s | master passed | | +1 :green_heart: | compile | 3m 19s | master passed | | +1 :green_heart: | checkstyle | 1m 14s | master passed | | +1 :green_heart: | spotbugs | 2m 8s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 51s | the patch passed | | +1 :green_heart: | compile | 3m 18s | the patch passed | | +1 :green_heart: | javac | 3m 18s | the patch passed | | +1 :green_heart: | checkstyle | 1m 12s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 59s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 49m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3049/11/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3049 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 79d5334b2cb3 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 85 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3049/11/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean commented on pull request #2941: HBASE-21674:Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from thrift1 server) to branch-1
YutSean commented on pull request #2941: URL: https://github.com/apache/hbase/pull/2941#issuecomment-800743363 @saintstack @Reidd The problem is still there. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25663) Make graceful_stop localhostname compare match even if fqdn
[
https://issues.apache.org/jira/browse/HBASE-25663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303026#comment-17303026
]
Hudson commented on HBASE-25663:
Results for branch branch-2.4
[build #74 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Make graceful_stop localhostname compare match even if fqdn
> ---
>
> Key: HBASE-25663
> URL: https://issues.apache.org/jira/browse/HBASE-25663
> Project: HBase
> Issue Type: Improvement
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Minor
>
> The hostname to operate on will be fqdn usually. Internal to the
> graceful_stop script it tries to check if we are running locally but the
> compare is of a fqdn to what 'hostname' returns; the latter usually returns
> name only. Allow name-only and fqdn match. While here cleanup doc around
> 'hostname' passed to graceful_stop.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25594) graceful_stop.sh fails to unload regions when ran at localhost
[
https://issues.apache.org/jira/browse/HBASE-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303027#comment-17303027
]
Hudson commented on HBASE-25594:
Results for branch branch-2.4
[build #74 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> graceful_stop.sh fails to unload regions when ran at localhost
> --
>
> Key: HBASE-25594
> URL: https://issues.apache.org/jira/browse/HBASE-25594
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 1.4.13
>Reporter: Javier Akira Luca de Tena
>Assignee: Javier Akira Luca de Tena
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> We usually use graceful_stop.sh from the Master to restart RegionServers.
> However, in some scenarios we may not have privileges to restart remote
> RegionServers (it uses ssh).
> But we can still use graceful_stop.sh on the same host we want to restart.
> In order to detect the execution at localhost, graceful_stop.sh uses
> /bin/hostname.
>
> [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/bin/graceful_stop.sh#L106-L110]
> When RegionMover strips the host to not include it in the list of target
> hosts, we filter it out by checking all RegionServer hosts in the cluster:
>
> [https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L382-L384]
>
> [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L692]
> But the list of RegionServer hosts returned by Admin#getRegionServers are
> FDQN, while the hostname provided from graceful_stop.sh is not FDQN, making
> the comparison fail.
> Same happens for branch-1 region_mover.rb, which is the place I reproduced in
> my environment:
> [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L305]
> [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L175]
>
> [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L186-L192]
>
> This can be fixed just by using "/bin/hostname -f" in the graceful_stop.sh
> script.
> Will provide patch soon.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25374) Make REST Client connection and socket time out configurable
[
https://issues.apache.org/jira/browse/HBASE-25374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303025#comment-17303025
]
Hudson commented on HBASE-25374:
Results for branch branch-2.4
[build #74 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/74/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Make REST Client connection and socket time out configurable
>
>
> Key: HBASE-25374
> URL: https://issues.apache.org/jira/browse/HBASE-25374
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Affects Versions: 3.0.0-alpha-1, 2.3.4, 2.5.0, 2.4.3
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3
>
>
> Currently there is no way to configure connection and socket timeout for REST
> client,
> https://github.com/apache/hbase/blob/7d0a687e5798a2f4ca3190b409169f7e17a75b34/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/Client.java#L102
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25621) Balancer should check region plan source to avoid misplace region groups
[
https://issues.apache.org/jira/browse/HBASE-25621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303006#comment-17303006
]
Hudson commented on HBASE-25621:
Results for branch branch-2.2
[build #191 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/191/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/191//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/191//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/191//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/191//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Balancer should check region plan source to avoid misplace region groups
>
>
> Key: HBASE-25621
> URL: https://issues.apache.org/jira/browse/HBASE-25621
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.3
>
>
> Currently, balancer generates all group plans firstly, and then execute
> overall plans. When there are table regions movement between groups but after
> the groups plans generating and before executing, balancer may misplace the
> region to the old group. But this is not serious problem, next balancer will
> fix it.
> If we add a check before executing balance plans, we can avoid the problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25595) TestLruBlockCache.testBackgroundEvictionThread is flaky
[
https://issues.apache.org/jira/browse/HBASE-25595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303005#comment-17303005
]
Hudson commented on HBASE-25595:
Results for branch branch-2.2
[build #191 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/191/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/191//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/191//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/191//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/191//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> TestLruBlockCache.testBackgroundEvictionThread is flaky
> ---
>
> Key: HBASE-25595
> URL: https://issues.apache.org/jira/browse/HBASE-25595
> Project: HBase
> Issue Type: Bug
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.3
>
>
> As the test result in
> [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2700/2/testReport/org.apache.hadoop.hbase.io.hfile/TestLruBlockCache/precommit_checks___yetus_jdk11_hadoop3_checks___testBackgroundEvictionThread/]
> It fails because background EvictionThread used the weak reference of
> LRUBlockCache, when the cache object is cleared by garbage collector, NO
> block eviction will be triggered by LruBlockCache#runEviction().
> .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-20589) Don't need to assign meta to a new RS when standby master become active
[
https://issues.apache.org/jira/browse/HBASE-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302987#comment-17302987
]
Hudson commented on HBASE-20589:
Results for branch branch-1
[build #97 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/97/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/97//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/97//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/97//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Don't need to assign meta to a new RS when standby master become active
> ---
>
> Key: HBASE-20589
> URL: https://issues.apache.org/jira/browse/HBASE-20589
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: HBASE-20589.branch-2.0.patch,
> HBASE-20589.master.001.patch, HBASE-20589.master.002.patch,
> HBASE-20589.master.003.patch, HBASE-20589.master.003.patch,
> HBASE-20589.master.004.patch, HBASE-20589.master.005.patch,
> HBASE-20589.master.006.patch, HBASE-20589.master.007.patch,
> HBASE-20589.master.008.patch, HBASE-20589.master.008.patch,
> HBASE-20589.master.009.patch
>
>
> I found this problem when I write ut for HBASE-20569. Now the master
> finishActiveMasterInitialization introduce a new
> RecoverMetaProcedure(HBASE-18261) and it has a sub procedure AssignProcedure.
> AssignProcedure will skip assign a region when regions state is OPEN and
> server is online. But for the new regiog state node is created with state
> OFFLINE. So it will assign the meta to a new RS. And kill the old RS when old
> RS report to master. This will make the master initialization cost a long
> time. I will attatch a ut to show this. FYI [~stack]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24395) ServerName#getHostname() is case sensitive
[
https://issues.apache.org/jira/browse/HBASE-24395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302986#comment-17302986
]
Hudson commented on HBASE-24395:
Results for branch branch-1
[build #97 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/97/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/97//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/97//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-1/97//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> ServerName#getHostname() is case sensitive
> --
>
> Key: HBASE-24395
> URL: https://issues.apache.org/jira/browse/HBASE-24395
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer
>Affects Versions: 1.3.1
>Reporter: Bo Cui
>Assignee: Bo Cui
>Priority: Major
> Fix For: 1.7.0
>
> Attachments: HBase-24395.patch, image-2020-05-18-17-42-57-119.png
>
>
> ServerName calss，the getServerName(String hostName, int port, long
> startcode),equals and compareTo are case insensitive, but getHostname() is
> case sensitive.
> if hostName is HOSTNAME1, ServerName is hostname1,1,1589615319931, and
> getHostname() returns HOSTNAME1.
> and then BaseLoadBalancer#retainAssignment() uses ServerName#getHostname(),
> all keys of serversByHostname are
> upperCase(HOSTNAME1,HOSTNAME2,HOSTNAME3,HOSTNAME4...) from
> ServerManager#createDestinationServersList, but oldServerName.getHostname()
> is lowerCase(hostname1,hostname2,hostname3...) from walLog dir.
> !image-2020-05-18-17-42-57-119.png!
> and finally...all region of old ServerName will be assigned to random hosts
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache-HBase commented on pull request #1708: URL: https://github.com/apache/hbase/pull/1708#issuecomment-800699682 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 44s | master passed | | +1 :green_heart: | compile | 3m 4s | master passed | | +1 :green_heart: | shadedjars | 8m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 12m 8s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 45s | the patch passed | | +1 :green_heart: | compile | 3m 39s | the patch passed | | +1 :green_heart: | javac | 3m 39s | the patch passed | | +1 :green_heart: | shadedjars | 9m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 11m 5s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 392m 53s | root in the patch failed. | | | | 458m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/12/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1708 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux fbea2160d07c 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/12/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/12/testReport/ | | Max. process+thread count | 4153 (vs. ulimit of 3) | | modules | C: hbase-logging hbase-common hbase-metrics-api hbase-metrics hbase-hadoop-compat hbase-client hbase-zookeeper hbase-replication hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-testing-util hbase-thrift hbase-shell hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-shaded hbase-shaded/hbase-shaded-client hbase-shaded/hbase-shaded-client-byo-hadoop hbase-shaded/hbase-shaded-mapreduce hbase-hbtop hbase-assembly hbase-shaded/hbase-shaded-testing-util hbase-shaded/hbase-shaded-testing-util-tester hbase-shaded/hbase-shaded-check-invariants hbase-shaded/hbase-shaded-with-hadoop-check-invariants hbase-archetypes/hbase-client-project hbase-archetypes/hbase-shaded-client-project . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/12/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25668) TestCurrentHourProvider fails 100% in branch-2.3
[ https://issues.apache.org/jira/browse/HBASE-25668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-25668: -- Attachment: screenshot-1.png > TestCurrentHourProvider fails 100% in branch-2.3 > > > Key: HBASE-25668 > URL: https://issues.apache.org/jira/browse/HBASE-25668 > Project: HBase > Issue Type: Sub-task > Components: flakies >Affects Versions: 2.3.4 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: 2.3.5 > > Attachments: image-2021-03-16-13-34-29-412.png, screenshot-1.png > > > !image-2021-03-16-13-34-29-412.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25668) TestCurrentHourProvider fails 100% in branch-2.3
[ https://issues.apache.org/jira/browse/HBASE-25668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302975#comment-17302975 ] Michael Stack commented on HBASE-25668: --- After revert, flakies shows 2.3 as passing for first time in a long time. !screenshot-1.png! > TestCurrentHourProvider fails 100% in branch-2.3 > > > Key: HBASE-25668 > URL: https://issues.apache.org/jira/browse/HBASE-25668 > Project: HBase > Issue Type: Sub-task > Components: flakies >Affects Versions: 2.3.4 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: 2.3.5 > > Attachments: image-2021-03-16-13-34-29-412.png, screenshot-1.png > > > !image-2021-03-16-13-34-29-412.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3056: HBASE-25669 Fix typo of hbase.mob.compaction.chore.period in the docs
Apache-HBase commented on pull request #3056: URL: https://github.com/apache/hbase/pull/3056#issuecomment-800676505 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 51s | master passed | | +0 :ok: | refguide | 3m 27s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 3m 20s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. | | | | 16m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3056/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3056 | | Optional Tests | dupname asflicense refguide | | uname | Linux b1b0c42f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3056/1/artifact/yetus-general-check/output/branch-site/book.html | | refguide | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3056/1/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3056/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3056: HBASE-25669 Fix typo of hbase.mob.compaction.chore.period in the docs
Apache-HBase commented on pull request #3056: URL: https://github.com/apache/hbase/pull/3056#issuecomment-800670580 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] nkalmar opened a new pull request #3056: HBASE-25669 Fix typo of hbase.mob.compaction.chore.period in the docs
nkalmar opened a new pull request #3056: URL: https://github.com/apache/hbase/pull/3056 This was confusing for me, as I managed to use the wrong configuration. I will check the other branches, as I was working with 2.2 when noticing this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25669) Typo in docs vs code: hbase.mob.[file.]compaction.chore.period
[ https://issues.apache.org/jira/browse/HBASE-25669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Kalmár updated HBASE-25669: --- Description: I might be missing something here, but after trying to disable mob compaction chore, and failed at it by setting hbase.mob.file.compaction.chore.period to 0, I realized docs and the code differ. Code: https://github.com/apache/hbase/search?q=hbase.mob.compaction.chore.period Docs: https://github.com/apache/hbase/search?q=hbase.mob.file.compaction.chore.period Looks like only a portion of the docs, still confusing. was: I might be missing something here, but after trying to disable mob compaction chore, and failed at it by setting hbase.mob.file.compaction.chore.period to 0, I realized docs and the code differ. Code: https://github.com/apache/hbase/search?q=hbase.mob.compaction.chore.period Docs: https://github.com/apache/hbase/search?q=hbase.mob.file.compaction.chore.period > Typo in docs vs code: hbase.mob.[file.]compaction.chore.period > -- > > Key: HBASE-25669 > URL: https://issues.apache.org/jira/browse/HBASE-25669 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1 >Reporter: Norbert Kalmár >Assignee: Norbert Kalmár >Priority: Major > > I might be missing something here, but after trying to disable mob compaction > chore, and failed at it by setting hbase.mob.file.compaction.chore.period to > 0, I realized docs and the code differ. > Code: > https://github.com/apache/hbase/search?q=hbase.mob.compaction.chore.period > Docs: > https://github.com/apache/hbase/search?q=hbase.mob.file.compaction.chore.period > Looks like only a portion of the docs, still confusing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25669) Typo in docs vs code: hbase.mob.[file.]compaction.chore.period
[ https://issues.apache.org/jira/browse/HBASE-25669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Kalmár updated HBASE-25669: --- Affects Version/s: 3.0.0-alpha-1 > Typo in docs vs code: hbase.mob.[file.]compaction.chore.period > -- > > Key: HBASE-25669 > URL: https://issues.apache.org/jira/browse/HBASE-25669 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1 >Reporter: Norbert Kalmár >Assignee: Norbert Kalmár >Priority: Major > > I might be missing something here, but after trying to disable mob compaction > chore, and failed at it by setting hbase.mob.file.compaction.chore.period to > 0, I realized docs and the code differ. > Code: > https://github.com/apache/hbase/search?q=hbase.mob.compaction.chore.period > Docs: > https://github.com/apache/hbase/search?q=hbase.mob.file.compaction.chore.period -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25669) Typo in docs vs code: hbase.mob.[file.]compaction.chore.period
[ https://issues.apache.org/jira/browse/HBASE-25669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Kalmár reassigned HBASE-25669: -- Assignee: Norbert Kalmár > Typo in docs vs code: hbase.mob.[file.]compaction.chore.period > -- > > Key: HBASE-25669 > URL: https://issues.apache.org/jira/browse/HBASE-25669 > Project: HBase > Issue Type: Bug >Reporter: Norbert Kalmár >Assignee: Norbert Kalmár >Priority: Major > > I might be missing something here, but after trying to disable mob compaction > chore, and failed at it by setting hbase.mob.file.compaction.chore.period to > 0, I realized docs and the code differ. > Code: > https://github.com/apache/hbase/search?q=hbase.mob.compaction.chore.period > Docs: > https://github.com/apache/hbase/search?q=hbase.mob.file.compaction.chore.period -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25669) Typo in docs vs code: hbase.mob.[file.]compaction.chore.period
Norbert Kalmár created HBASE-25669: -- Summary: Typo in docs vs code: hbase.mob.[file.]compaction.chore.period Key: HBASE-25669 URL: https://issues.apache.org/jira/browse/HBASE-25669 Project: HBase Issue Type: Bug Reporter: Norbert Kalmár I might be missing something here, but after trying to disable mob compaction chore, and failed at it by setting hbase.mob.file.compaction.chore.period to 0, I realized docs and the code differ. Code: https://github.com/apache/hbase/search?q=hbase.mob.compaction.chore.period Docs: https://github.com/apache/hbase/search?q=hbase.mob.file.compaction.chore.period -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25668) TestCurrentHourProvider fails 100% in branch-2.3
[
https://issues.apache.org/jira/browse/HBASE-25668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-25668.
---
Resolution: Fixed
Pushed below on branch-2.3.
{code}
commit bd250f57f7b0cefe78b6f50f99d98e7c8ad4d2f5 (HEAD -> 2.3.clean,
origin/branch-2.3)
Author: stack
Date: Tue Mar 16 15:17:43 2021 -0700
HBASE-25668 TestCurrentHourProvider fails 100% in branch-2.3"
This commit removes the two previous minor debug commits. I then also
includes revert of:
1ba49309cc HBASE-24898 Addendum to deal with Daylight Saving Time (#2318)
28cb121c36 HBASE-24898 Use EnvironmentEdge.currentTime() instead of
System.currentTimeMillis() in CurrentHourProvider
{code}
> TestCurrentHourProvider fails 100% in branch-2.3
>
>
> Key: HBASE-25668
> URL: https://issues.apache.org/jira/browse/HBASE-25668
> Project: HBase
> Issue Type: Sub-task
> Components: flakies
>Affects Versions: 2.3.4
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Minor
> Fix For: 2.3.5
>
> Attachments: image-2021-03-16-13-34-29-412.png
>
>
> !image-2021-03-16-13-34-29-412.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25668) TestCurrentHourProvider fails 100% in branch-2.3
[ https://issues.apache.org/jira/browse/HBASE-25668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302944#comment-17302944 ] Michael Stack commented on HBASE-25668: --- I just reverted my debug and the two above commits for now. Resolving. We can open new subtask to discuss fix for branch-2.3. > TestCurrentHourProvider fails 100% in branch-2.3 > > > Key: HBASE-25668 > URL: https://issues.apache.org/jira/browse/HBASE-25668 > Project: HBase > Issue Type: Sub-task > Components: flakies >Affects Versions: 2.3.4 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: 2.3.5 > > Attachments: image-2021-03-16-13-34-29-412.png > > > !image-2021-03-16-13-34-29-412.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25668) TestCurrentHourProvider fails 100% in branch-2.3
[ https://issues.apache.org/jira/browse/HBASE-25668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302941#comment-17302941 ] Michael Stack commented on HBASE-25668: --- I was just going to revert these two changes from branch-2.3 unless you have a better idea [~filtertip] ? Here is what I would revert: 1ba49309cc HBASE-24898 Addendum to deal with Daylight Saving Time (#2318) 28cb121c36 HBASE-24898 Use EnvironmentEdge.currentTime() instead of System.currentTimeMillis() in CurrentHourProvider 2.3 currently has some debug that prints out config for each timezone if interested. > TestCurrentHourProvider fails 100% in branch-2.3 > > > Key: HBASE-25668 > URL: https://issues.apache.org/jira/browse/HBASE-25668 > Project: HBase > Issue Type: Sub-task > Components: flakies >Affects Versions: 2.3.4 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: 2.3.5 > > Attachments: image-2021-03-16-13-34-29-412.png > > > !image-2021-03-16-13-34-29-412.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25668) TestCurrentHourProvider fails 100% in branch-2.3
[ https://issues.apache.org/jira/browse/HBASE-25668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302937#comment-17302937 ] Michael Stack commented on HBASE-25668: --- This is what messes up the unit test when run over on the apache test machine: 2021-03-16 21:47:39,352 INFO [Time-limited test] compactions.TestCurrentHourProvider(68): Timezone=sun.util.calendar.ZoneInfo[id="America/Dawson",offset=-2520,dstSavings=0,useDaylight=false,transitions=94,lastRule=null] java.lang.AssertionError: Timezone=sun.util.calendar.ZoneInfo[id="America/Dawson",offset=-2520,dstSavings=0,useDaylight=false,transitions=94,lastRule=null] expected:<11> but was:<10> at org.apache.hadoop.hbase.regionserver.compactions.TestCurrentHourProvider.testWithEnvironmentEdge(TestCurrentHourProvider.java:69) When it succeeds, it is like this: 2021-03-16 14:40:40,580 INFO [Time-limited test] compactions.TestCurrentHourProvider(68): Timezone=sun.util.calendar.ZoneInfo[id="America/Dawson",offset=-2880,dstSavings=360,useDaylight=true, transitions=127,lastRule=java.util.SimpleTimeZone[id=America/Dawson,offset=-2880,dstSavings=360,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=720, startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=720,endTimeMode=0]] The JVMs are saying different things about the America/Dawson timezone. One is linux and other is mac. > TestCurrentHourProvider fails 100% in branch-2.3 > > > Key: HBASE-25668 > URL: https://issues.apache.org/jira/browse/HBASE-25668 > Project: HBase > Issue Type: Sub-task > Components: flakies >Affects Versions: 2.3.4 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: 2.3.5 > > Attachments: image-2021-03-16-13-34-29-412.png > > > !image-2021-03-16-13-34-29-412.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Apache-HBase commented on pull request #3001: URL: https://github.com/apache/hbase/pull/3001#issuecomment-800639694 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 8s | master passed | | +1 :green_heart: | compile | 1m 43s | master passed | | +1 :green_heart: | shadedjars | 8m 26s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 5s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 25s | the patch passed | | +1 :green_heart: | compile | 1m 34s | the patch passed | | +1 :green_heart: | javac | 1m 34s | the patch passed | | -1 :x: | shadedjars | 6m 22s | patch has 10 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 45s | hbase-common in the patch passed. | | -1 :x: | unit | 226m 36s | hbase-server in the patch failed. | | | | 261m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3001 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8cb737d4bec5 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/testReport/ | | Max. process+thread count | 3136 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25668) TestCurrentHourProvider fails 100% in branch-2.3
[
https://issues.apache.org/jira/browse/HBASE-25668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302906#comment-17302906
]
Michael Stack commented on HBASE-25668:
---
Pushed on branch-2.3 some debug since it fails reliably on apache -- in the
flakies test each time around even -- but not locally:
{code}
commit 74a597fcfffa00907cbc04753b9f318ab659 (HEAD -> 2.3, origin/branch-2.3)
Author: stack
Date: Tue Mar 16 14:24:10 2021 -0700
HBASE-25668 TestCurrentHourProvider fails 100% in branch-2.3
Debug...
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCurrentHourProvider.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCurrentHourProvider.java
index 6df6b9b570..f7141e7848 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCurrentHourProvider.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCurrentHourProvider.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -18,7 +18,6 @@
package org.apache.hadoop.hbase.regionserver.compactions;
import static org.junit.Assert.assertEquals;
-
import java.util.Date;
import java.util.TimeZone;
import org.apache.hadoop.hbase.HBaseClassTestRule;
@@ -63,7 +62,7 @@ public class TestCurrentHourProvider {
CurrentHourProvider.getCurrentHour() - 2 :
CurrentHourProvider.getCurrentHour() - 1;
}
- assertEquals(11, hour11);
+ assertEquals("Timezone=" + timezone, 11, hour11);
// set a time represent hour 15
long deltaFor15 = TimeZone.getDefault().getRawOffset() - 2880;
@@ -76,7 +75,7 @@ public class TestCurrentHourProvider {
CurrentHourProvider.getCurrentHour() - 2 :
CurrentHourProvider.getCurrentHour() - 1;
}
- assertEquals(15, hour15);
+ assertEquals("Timezone=" + timezone, 15, hour15);
}
}
}
{code}
> TestCurrentHourProvider fails 100% in branch-2.3
>
>
> Key: HBASE-25668
> URL: https://issues.apache.org/jira/browse/HBASE-25668
> Project: HBase
> Issue Type: Sub-task
> Components: flakies
>Affects Versions: 2.3.4
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Minor
> Fix For: 2.3.5
>
> Attachments: image-2021-03-16-13-34-29-412.png
>
>
> !image-2021-03-16-13-34-29-412.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] huaxiangsun commented on pull request #3052: HBASE-25639 meta replica state is not respected during active master …
huaxiangsun commented on pull request #3052: URL: https://github.com/apache/hbase/pull/3052#issuecomment-800616070 Yeah, I am trying it out at a cluster. Will report back. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on pull request #3052: HBASE-25639 meta replica state is not respected during active master …
huaxiangsun commented on pull request #3052: URL: https://github.com/apache/hbase/pull/3052#issuecomment-800615749 > LGTM. Is this a backport then? Did you try it @huaxiangsun ? It is not really a backport. The 2.4/master way of handling of meta replica is different. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] huaxiangsun commented on a change in pull request #3052: HBASE-25639 meta replica state is not respected during active master …
huaxiangsun commented on a change in pull request #3052:
URL: https://github.com/apache/hbase/pull/3052#discussion_r595548965
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java
##
@@ -61,25 +60,25 @@ void assignMetaReplicas()
throw new IllegalStateException("hbase:meta must be initialized first
before we can " +
"assign out its replicas");
}
-ServerName metaServername =
MetaTableLocator.getMetaRegionLocation(this.master.getZooKeeper());
+
for (int i = 1; i < numReplicas; i++) {
// Get current meta state for replica from zk.
- RegionState metaState =
MetaTableLocator.getMetaRegionState(master.getZooKeeper(), i);
RegionInfo hri = RegionReplicaUtil.getRegionInfoForReplica(
- RegionInfoBuilder.FIRST_META_REGIONINFO, i);
- LOG.debug(hri.getRegionNameAsString() + " replica region state from
zookeeper=" + metaState);
- if (metaServername.equals(metaState.getServerName())) {
-metaState = null;
-LOG.info(hri.getRegionNameAsString() +
- " old location is same as current hbase:meta location; setting
location as null...");
- }
Review comment:
I think it is a bad optimization. What it tries to do is to avoid
assigning meta replica to the same region server where primary region is
located. This part of old logic is untested as there is no unassign procedure
for the meta replica region. It is ok to keep the original assignment and let
balancer to move them apart.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3055: HBASE-25656 Backport to branch-2.2: [HBASE-25548 Optionally allow sna…
Apache-HBase commented on pull request #3055: URL: https://github.com/apache/hbase/pull/3055#issuecomment-800613182 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ branch-2.2 Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 23s | branch-2.2 passed | | +1 :green_heart: | compile | 1m 57s | branch-2.2 passed | | +1 :green_heart: | checkstyle | 4m 38s | branch-2.2 passed | | +0 :ok: | refguide | 16m 14s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 4m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 46s | branch-2.2 passed | | +0 :ok: | spotbugs | 3m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 17m 12s | branch-2.2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 3s | the patch passed | | +1 :green_heart: | compile | 2m 7s | the patch passed | | +1 :green_heart: | cc | 2m 7s | the patch passed | | +1 :green_heart: | javac | 2m 7s | the patch passed | | -1 :x: | checkstyle | 0m 40s | hbase-client: The patch generated 3 new + 270 unchanged - 0 fixed = 273 total (was 270) | | -1 :x: | checkstyle | 2m 17s | root: The patch generated 3 new + 298 unchanged - 0 fixed = 301 total (was 298) | | -1 :x: | rubocop | 0m 10s | The patch generated 3 new + 370 unchanged - 1 fixed = 373 total (was 371) | | -1 :x: | whitespace | 0m 0s | The patch has 25 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +0 :ok: | refguide | 5m 22s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 3m 55s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 27m 38s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | hbaseprotoc | 5m 28s | the patch passed | | -1 :x: | javadoc | 0m 25s | hbase-client generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2) | | -1 :x: | javadoc | 2m 18s | root generated 1 new + 6 unchanged - 0 fixed = 7 total (was 6) | | -1 :x: | findbugs | 1m 28s | hbase-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -1 :x: | findbugs | 9m 16s | root generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | unit | 138m 52s | root in the patch passed. | | +1 :green_heart: | asflicense | 2m 40s | The patch does not generate ASF License warnings. | | | | 277m 25s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-client | | | Useless object stored in variable snapshotProps of method org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.createSnapshotDesc(SnapshotProtos$SnapshotDescription) At ProtobufUtil.java:snapshotProps of method org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.createSnapshotDesc(SnapshotProtos$SnapshotDescription) At ProtobufUtil.java:[line 2946] | | FindBugs | module:root | | | Useless object stored in variable snapshotProps of method org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.createSnapshotDesc(SnapshotProtos$SnapshotDescription) At ProtobufUtil.java:snapshotProps of method org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.createSnapshotDesc(SnapshotProtos$SnapshotDescription) At ProtobufUtil.java:[line 2946] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3055/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3055 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
[GitHub] [hbase] huaxiangsun commented on a change in pull request #3052: HBASE-25639 meta replica state is not respected during active master …
huaxiangsun commented on a change in pull request #3052:
URL: https://github.com/apache/hbase/pull/3052#discussion_r595547253
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -1018,7 +1018,7 @@ private void
finishActiveMasterInitialization(MonitoredTask status)
RegionState rs = this.assignmentManager.getRegionStates().
getRegionState(RegionInfoBuilder.FIRST_META_REGIONINFO);
LOG.info("hbase:meta {}", rs);
-if (rs != null && rs.isOffline()) {
+if ((rs == null) || (rs != null && rs.isOffline())) {
Review comment:
When it is a new meta replica config, there is no meta region server
node at zookeeper. In the old way, it will create a RegionState with offline
state. In this new way, there will be no RegoinState created. So that is why rs
== null is added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache-HBase commented on pull request #1708: URL: https://github.com/apache/hbase/pull/1708#issuecomment-800585862 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 11s | master passed | | +1 :green_heart: | compile | 2m 52s | master passed | | +1 :green_heart: | shadedjars | 6m 56s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 13m 1s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 14s | the patch passed | | +1 :green_heart: | compile | 2m 56s | the patch passed | | +1 :green_heart: | javac | 2m 56s | the patch passed | | +1 :green_heart: | shadedjars | 7m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 13m 1s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 170m 22s | root in the patch failed. | | | | 235m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/12/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1708 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a4933970c0ba 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/12/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/12/testReport/ | | Max. process+thread count | 5516 (vs. ulimit of 3) | | modules | C: hbase-logging hbase-common hbase-metrics-api hbase-metrics hbase-hadoop-compat hbase-client hbase-zookeeper hbase-replication hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-testing-util hbase-thrift hbase-shell hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-shaded hbase-shaded/hbase-shaded-client hbase-shaded/hbase-shaded-client-byo-hadoop hbase-shaded/hbase-shaded-mapreduce hbase-hbtop hbase-assembly hbase-shaded/hbase-shaded-testing-util hbase-shaded/hbase-shaded-testing-util-tester hbase-shaded/hbase-shaded-check-invariants hbase-shaded/hbase-shaded-with-hadoop-check-invariants hbase-archetypes/hbase-client-project hbase-archetypes/hbase-shaded-client-project . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/12/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25668) TestCurrentHourProvider fails 100% in branch-2.3
Michael Stack created HBASE-25668: - Summary: TestCurrentHourProvider fails 100% in branch-2.3 Key: HBASE-25668 URL: https://issues.apache.org/jira/browse/HBASE-25668 Project: HBase Issue Type: Sub-task Components: flakies Affects Versions: 2.3.4 Reporter: Michael Stack Assignee: Michael Stack Fix For: 2.3.5 Attachments: image-2021-03-16-13-34-29-412.png !image-2021-03-16-13-34-29-412.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24898) Use EnvironmentEdge.currentTime() instead of System.currentTimeMillis() in CurrentHourProvider
[ https://issues.apache.org/jira/browse/HBASE-24898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302877#comment-17302877 ] Michael Stack commented on HBASE-24898: --- The 2.3 backport fails 100% of the time. Let me address in subtask. > Use EnvironmentEdge.currentTime() instead of System.currentTimeMillis() in > CurrentHourProvider > -- > > Key: HBASE-24898 > URL: https://issues.apache.org/jira/browse/HBASE-24898 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 2.2.6, 2.3.2 > > > In order to control the return value of getCurrentHour used by unit test. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Apache-HBase commented on pull request #3001: URL: https://github.com/apache/hbase/pull/3001#issuecomment-800584080 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 7s | master passed | | +1 :green_heart: | compile | 1m 34s | master passed | | +1 :green_heart: | shadedjars | 6m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 1m 32s | the patch passed | | +1 :green_heart: | javac | 1m 32s | the patch passed | | -1 :x: | shadedjars | 5m 28s | patch has 10 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 35s | hbase-common in the patch passed. | | -1 :x: | unit | 144m 38s | hbase-server in the patch failed. | | | | 175m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3001 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a58b905691b4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | Default Java | AdoptOpenJDK-11.0.10+9 | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/testReport/ | | Max. process+thread count | 4936 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25667) Remove RSGroup test addition made in parent; depends on functionality not in old branches
[
https://issues.apache.org/jira/browse/HBASE-25667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-25667.
---
Fix Version/s: 2.4.3
2.3.5
2.5.0
Assignee: Michael Stack
Resolution: Fixed
I just reverted the test added by the parent... and pushed.
{code}
commit e70f5ac37549e82099c45f45b7ea406d1fcc16a3 (HEAD -> 2.3, origin/branch-2.3)
Author: stack
Date: Tue Mar 16 13:17:13 2021 -0700
HBASE-25667 Remove RSGroup test addition made in parent; depends on
functionality not in old branches
diff --git a/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
b/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
index 694305f4e9..a5fd0ca66c 100644
--- a/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
+++ b/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
@@ -115,16 +115,8 @@ module Hbase
assert_not_nil(group)
assert_equal(ns_table_name, group.getTables.iterator.next.toString)
- ns_table_name2 = 'test_namespace:test_ns_table2'
- @shell.command(:create, ns_table_name2, 'f')
-
-
assert_true(@admin.listTablesInRSGroup(group_name).contains(org.apache.hadoop.hbase.TableName.valueOf(ns_table_name2)))
- assert_equal(2, @admin.listTablesInRSGroup(group_name).count)
-
@shell.command(:disable, ns_table_name)
@shell.command(:drop, ns_table_name)
- @shell.command(:disable, ns_table_name2)
- @shell.command(:drop, ns_table_name2)
@shell.command(:drop_namespace, namespace_name)
remove_rsgroup(group_name)
end
{code}
> Remove RSGroup test addition made in parent; depends on functionality not in
> old branches
> -
>
> Key: HBASE-25667
> URL: https://issues.apache.org/jira/browse/HBASE-25667
> Project: HBase
> Issue Type: Sub-task
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 2.5.0, 2.3.5, 2.4.3
>
>
> TestRSGroupShell fails 100% of the time. See
> https://ci-hadoop.apache.org/view/HBase/job/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.3/lastSuccessfulBuild/artifact/output/dashboard.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25667) Remove RSGroup test addition made in parent; depends on functionality not in old branches
[ https://issues.apache.org/jira/browse/HBASE-25667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-25667: -- Description: TestRSGroupShell fails 100% of the time. See https://ci-hadoop.apache.org/view/HBase/job/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.3/lastSuccessfulBuild/artifact/output/dashboard.html > Remove RSGroup test addition made in parent; depends on functionality not in > old branches > - > > Key: HBASE-25667 > URL: https://issues.apache.org/jira/browse/HBASE-25667 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > > TestRSGroupShell fails 100% of the time. See > https://ci-hadoop.apache.org/view/HBase/job/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.3/lastSuccessfulBuild/artifact/output/dashboard.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25667) Remove RSGroup test addition made in parent; depends on functionality not in old branches
[ https://issues.apache.org/jira/browse/HBASE-25667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-25667: -- Environment: (was: TestRSGroupShell fails 100% of the time. See https://ci-hadoop.apache.org/view/HBase/job/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.3/lastSuccessfulBuild/artifact/output/dashboard.html) > Remove RSGroup test addition made in parent; depends on functionality not in > old branches > - > > Key: HBASE-25667 > URL: https://issues.apache.org/jira/browse/HBASE-25667 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25667) Remove RSGroup test addition made in parent; depends on functionality not in old branches
Michael Stack created HBASE-25667: - Summary: Remove RSGroup test addition made in parent; depends on functionality not in old branches Key: HBASE-25667 URL: https://issues.apache.org/jira/browse/HBASE-25667 Project: HBase Issue Type: Sub-task Environment: TestRSGroupShell fails 100% of the time. See https://ci-hadoop.apache.org/view/HBase/job/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.3/lastSuccessfulBuild/artifact/output/dashboard.html Reporter: Michael Stack -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25497) move_namespaces_rsgroup should change hbase.rsgroup.name config in NamespaceDescriptor
[ https://issues.apache.org/jira/browse/HBASE-25497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302862#comment-17302862 ] Michael Stack commented on HBASE-25497: --- The backport here depends on Admin#listTablesInRSGroup which does not exist in branch-2 hbase. Let me file a subtask to undo the reliance which is breaking RSGroup tests. > move_namespaces_rsgroup should change hbase.rsgroup.name config in > NamespaceDescriptor > -- > > Key: HBASE-25497 > URL: https://issues.apache.org/jira/browse/HBASE-25497 > Project: HBase > Issue Type: Bug >Reporter: xijiawen >Assignee: xijiawen >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.2 > > > move_namespaces_rsgroup command just move tables in this namespace to new > rsgroup。when we create a table in this namespace，it deploy on old rs group。 > it maka some misunderstand for users。 > move_namespaces_rsgroup should make the tables deployed on new rsgroup -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on a change in pull request #3052: HBASE-25639 meta replica state is not respected during active master …
saintstack commented on a change in pull request #3052:
URL: https://github.com/apache/hbase/pull/3052#discussion_r595472685
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java
##
@@ -61,25 +60,25 @@ void assignMetaReplicas()
throw new IllegalStateException("hbase:meta must be initialized first
before we can " +
"assign out its replicas");
}
-ServerName metaServername =
MetaTableLocator.getMetaRegionLocation(this.master.getZooKeeper());
+
for (int i = 1; i < numReplicas; i++) {
// Get current meta state for replica from zk.
- RegionState metaState =
MetaTableLocator.getMetaRegionState(master.getZooKeeper(), i);
RegionInfo hri = RegionReplicaUtil.getRegionInfoForReplica(
- RegionInfoBuilder.FIRST_META_REGIONINFO, i);
- LOG.debug(hri.getRegionNameAsString() + " replica region state from
zookeeper=" + metaState);
- if (metaServername.equals(metaState.getServerName())) {
-metaState = null;
-LOG.info(hri.getRegionNameAsString() +
- " old location is same as current hbase:meta location; setting
location as null...");
- }
Review comment:
We don't need this handling anymore?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -1018,7 +1018,7 @@ private void
finishActiveMasterInitialization(MonitoredTask status)
RegionState rs = this.assignmentManager.getRegionStates().
getRegionState(RegionInfoBuilder.FIRST_META_REGIONINFO);
LOG.info("hbase:meta {}", rs);
-if (rs != null && rs.isOffline()) {
+if ((rs == null) || (rs != null && rs.isOffline())) {
Review comment:
No RegionState for replicas?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache-HBase commented on pull request #1708: URL: https://github.com/apache/hbase/pull/1708#issuecomment-800515831 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 1s | Shelldocs was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 42s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 35s | master passed | | +1 :green_heart: | compile | 8m 7s | master passed | | +1 :green_heart: | checkstyle | 1m 59s | master passed | | +1 :green_heart: | spotbugs | 23m 51s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 33s | the patch passed | | +1 :green_heart: | compile | 8m 10s | the patch passed | | +1 :green_heart: | javac | 8m 10s | root generated 0 new + 1560 unchanged - 2 fixed = 1560 total (was 1562) | | +1 :green_heart: | checkstyle | 1m 58s | root: The patch generated 0 new + 163 unchanged - 3 fixed = 163 total (was 166) | | +1 :green_heart: | shellcheck | 0m 2s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 52s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 17m 58s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 29m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 7m 5s | The patch does not generate ASF License warnings. | | | | 124m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/12/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1708 | | Optional Tests | dupname asflicense shellcheck shelldocs xml javac hadoopcheck compile spotbugs hbaseanti checkstyle | | uname | Linux f8a05e646820 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | Max. process+thread count | 141 (vs. ulimit of 3) | | modules | C: hbase-logging hbase-common hbase-metrics-api hbase-metrics hbase-hadoop-compat hbase-client hbase-zookeeper hbase-replication hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-testing-util hbase-thrift hbase-shell hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-shaded hbase-shaded/hbase-shaded-client hbase-shaded/hbase-shaded-client-byo-hadoop hbase-shaded/hbase-shaded-mapreduce hbase-hbtop hbase-assembly hbase-shaded/hbase-shaded-testing-util hbase-shaded/hbase-shaded-testing-util-tester hbase-shaded/hbase-shaded-check-invariants hbase-shaded/hbase-shaded-with-hadoop-check-invariants hbase-archetypes/hbase-client-project hbase-archetypes/hbase-shaded-client-project . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/12/console | | versions | git=2.17.1 maven=3.6.3 shellcheck=0.4.6 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #3049: HBASE-25643 The delayed FlushRegionEntry should be removed when we ne…
saintstack commented on a change in pull request #3049:
URL: https://github.com/apache/hbase/pull/3049#discussion_r595446538
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
##
@@ -127,20 +127,22 @@ public MemStoreFlusher(final Configuration conf,
this.blockingWaitTime = conf.getInt("hbase.hstore.blockingWaitTime",
9);
int handlerCount = conf.getInt("hbase.hstore.flusher.count", 2);
-if (handlerCount < 1) {
- LOG.warn("hbase.hstore.flusher.count was configed to {} which is less
than 1, corrected to 1",
- handlerCount);
- handlerCount = 1;
+if (server != null) {
+ if (handlerCount < 1) {
+LOG.warn("hbase.hstore.flusher.count was configed to {} which is less
than 1, "
++ "corrected to 1", handlerCount);
+handlerCount = 1;
+ }
+ LOG.info("globalMemStoreLimit="
+ + TraditionalBinaryPrefix
+
.long2String(this.server.getRegionServerAccounting().getGlobalMemStoreLimit(),
"", 1)
+ + ", globalMemStoreLimitLowMark="
+ + TraditionalBinaryPrefix.long2String(
+
this.server.getRegionServerAccounting().getGlobalMemStoreLimitLowMark(), "", 1)
+ + ", Offheap="
+ + (this.server.getRegionServerAccounting().isOffheap()));
}
this.flushHandlers = new FlushHandler[handlerCount];
Review comment:
If making a new PR, do you want to move the handlerCount check for < 1
outside of the if server != null paren? Seems like a generally applicable good
test... no need to keep it exclusive to when the server is non-null?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25663) Make graceful_stop localhostname compare match even if fqdn
[
https://issues.apache.org/jira/browse/HBASE-25663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302769#comment-17302769
]
Hudson commented on HBASE-25663:
Results for branch branch-2
[build #201 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Make graceful_stop localhostname compare match even if fqdn
> ---
>
> Key: HBASE-25663
> URL: https://issues.apache.org/jira/browse/HBASE-25663
> Project: HBase
> Issue Type: Improvement
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Minor
>
> The hostname to operate on will be fqdn usually. Internal to the
> graceful_stop script it tries to check if we are running locally but the
> compare is of a fqdn to what 'hostname' returns; the latter usually returns
> name only. Allow name-only and fqdn match. While here cleanup doc around
> 'hostname' passed to graceful_stop.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Apache-HBase commented on pull request #3001: URL: https://github.com/apache/hbase/pull/3001#issuecomment-800504302 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 46s | master passed | | +1 :green_heart: | compile | 3m 57s | master passed | | +1 :green_heart: | checkstyle | 1m 31s | master passed | | +1 :green_heart: | spotbugs | 2m 49s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 36s | the patch passed | | +1 :green_heart: | compile | 4m 2s | the patch passed | | +1 :green_heart: | javac | 4m 2s | the patch passed | | -0 :warning: | checkstyle | 1m 5s | hbase-server: The patch generated 3 new + 1 unchanged - 1 fixed = 4 total (was 2) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 43s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 10s | the patch passed | ||| _ Other Tests _ | | -1 :x: | asflicense | 0m 27s | The patch generated 1 ASF License warnings. | | | | 51m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3001 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux bff6acf337ea 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | asflicense | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/artifact/yetus-general-check/output/patch-asflicense-problems.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/3/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25660) Print split policy in use on Region open (as well as split policy vitals)
[
https://issues.apache.org/jira/browse/HBASE-25660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302770#comment-17302770
]
Hudson commented on HBASE-25660:
Results for branch branch-2
[build #201 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Print split policy in use on Region open (as well as split policy vitals)
> -
>
> Key: HBASE-25660
> URL: https://issues.apache.org/jira/browse/HBASE-25660
> Project: HBase
> Issue Type: Improvement
>Reporter: Michael Stack
>Priority: Trivial
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> On Region open, add to the open message the split and flush policies in
> place; they are printed no where else. For split policy, list vitals like
> split size for the Region; for the likes of ConstantSizeSplitPolicy there can
> be high variance in size so dump out what we've figured.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25374) Make REST Client connection and socket time out configurable
[
https://issues.apache.org/jira/browse/HBASE-25374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302775#comment-17302775
]
Hudson commented on HBASE-25374:
Results for branch branch-2
[build #201 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Make REST Client connection and socket time out configurable
>
>
> Key: HBASE-25374
> URL: https://issues.apache.org/jira/browse/HBASE-25374
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Affects Versions: 3.0.0-alpha-1, 2.3.4, 2.5.0, 2.4.3
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3
>
>
> Currently there is no way to configure connection and socket timeout for REST
> client,
> https://github.com/apache/hbase/blob/7d0a687e5798a2f4ca3190b409169f7e17a75b34/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/Client.java#L102
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25662) Fix spotbugs warning in RoundRobinTableInputFormat
[
https://issues.apache.org/jira/browse/HBASE-25662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302768#comment-17302768
]
Hudson commented on HBASE-25662:
Results for branch branch-2
[build #201 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix spotbugs warning in RoundRobinTableInputFormat
> --
>
> Key: HBASE-25662
> URL: https://issues.apache.org/jira/browse/HBASE-25662
> Project: HBase
> Issue Type: Bug
> Components: findbugs
>Reporter: Pankaj Kumar
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3
>
>
> RoundRobinTableInputFormat.roundRobin(List) makes inefficient use of keySet
> iterator instead of entrySet iterator,
> https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3043/2/artifact/yetus-general-check/output/branch-spotbugs-root-warnings.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25594) graceful_stop.sh fails to unload regions when ran at localhost
[
https://issues.apache.org/jira/browse/HBASE-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302776#comment-17302776
]
Hudson commented on HBASE-25594:
Results for branch branch-2
[build #201 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> graceful_stop.sh fails to unload regions when ran at localhost
> --
>
> Key: HBASE-25594
> URL: https://issues.apache.org/jira/browse/HBASE-25594
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 1.4.13
>Reporter: Javier Akira Luca de Tena
>Assignee: Javier Akira Luca de Tena
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3
>
>
> We usually use graceful_stop.sh from the Master to restart RegionServers.
> However, in some scenarios we may not have privileges to restart remote
> RegionServers (it uses ssh).
> But we can still use graceful_stop.sh on the same host we want to restart.
> In order to detect the execution at localhost, graceful_stop.sh uses
> /bin/hostname.
>
> [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/bin/graceful_stop.sh#L106-L110]
> When RegionMover strips the host to not include it in the list of target
> hosts, we filter it out by checking all RegionServer hosts in the cluster:
>
> [https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L382-L384]
>
> [https://github.com/apache/hbase/blob/cfbae4d3a37e7ac4d795461c3e19406a2786838d/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java#L692]
> But the list of RegionServer hosts returned by Admin#getRegionServers are
> FDQN, while the hostname provided from graceful_stop.sh is not FDQN, making
> the comparison fail.
> Same happens for branch-1 region_mover.rb, which is the place I reproduced in
> my environment:
> [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L305]
> [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L175]
>
> [https://github.com/apache/hbase/blob/f9a91488b2c39320bed502619bf7adb765c79de6/bin/region_mover.rb#L186-L192]
>
> This can be fixed just by using "/bin/hostname -f" in the graceful_stop.sh
> script.
> Will provide patch soon.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25635) CandidateGenerator may miss some region balance actions
[
https://issues.apache.org/jira/browse/HBASE-25635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302767#comment-17302767
]
Hudson commented on HBASE-25635:
Results for branch branch-2
[build #201 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> CandidateGenerator may miss some region balance actions
> ---
>
> Key: HBASE-25635
> URL: https://issues.apache.org/jira/browse/HBASE-25635
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Affects Versions: 2.0.0
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3
>
> Attachments: 1614940188041-image.png
>
>
> {color:#172b4d}In codes of CandidateGenerator#getAction()，valid region index
> are greater or equal to zero. Currently only regions with greater than zero
> indexes can get balance actions.{color}
> {code:java}
> protected BaseLoadBalancer.Cluster.Action getAction(int fromServer, int
> fromRegion,
> int toServer, int toRegion) {
> if (fromServer < 0 || toServer < 0) {
> return BaseLoadBalancer.Cluster.NullAction;
> }
> if (fromRegion > 0 && toRegion > 0) {
> return new BaseLoadBalancer.Cluster.SwapRegionsAction(fromServer,
> fromRegion,
> toServer, toRegion);
> } else if (fromRegion > 0) {
> return new BaseLoadBalancer.Cluster.MoveRegionAction(fromRegion,
> fromServer, toServer);
> } else if (toRegion > 0) {
> return new BaseLoadBalancer.Cluster.MoveRegionAction(toRegion, toServer,
> fromServer);
> } else {
> return BaseLoadBalancer.Cluster.NullAction;
> }
> }{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25597) Add row info in Exception when cell size exceeds maxCellSize
[
https://issues.apache.org/jira/browse/HBASE-25597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302774#comment-17302774
]
Hudson commented on HBASE-25597:
Results for branch branch-2
[build #201 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add row info in Exception when cell size exceeds maxCellSize
>
>
> Key: HBASE-25597
> URL: https://issues.apache.org/jira/browse/HBASE-25597
> Project: HBase
> Issue Type: Improvement
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3
>
>
> When cell size exceeds maxCellSize(default is 10M), client will get a
> DoNotRetryIOException, code as below:
> {code:java}
> private void checkCellSizeLimit(final HRegion r, final Mutation m) throws
> IOException {
> if (r.maxCellSize > 0) {
> CellScanner cells = m.cellScanner();
> while (cells.advance()) {
> int size = PrivateCellUtil.estimatedSerializedSizeOf(cells.current());
> if (size > r.maxCellSize) {
> String msg = "Cell with size " + size + " exceeds limit of " +
> r.maxCellSize + " bytes";
> LOG.debug(msg);
> throw new DoNotRetryIOException(msg);
> }
> }
> }
> }
> {code}
> There is no row related information, which makes troubleshooting difficult.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25621) Balancer should check region plan source to avoid misplace region groups
[
https://issues.apache.org/jira/browse/HBASE-25621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302773#comment-17302773
]
Hudson commented on HBASE-25621:
Results for branch branch-2
[build #201 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Balancer should check region plan source to avoid misplace region groups
>
>
> Key: HBASE-25621
> URL: https://issues.apache.org/jira/browse/HBASE-25621
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.3
>
>
> Currently, balancer generates all group plans firstly, and then execute
> overall plans. When there are table regions movement between groups but after
> the groups plans generating and before executing, balancer may misplace the
> region to the old group. But this is not serious problem, next balancer will
> fix it.
> If we add a check before executing balance plans, we can avoid the problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25561) Added ignored test for async connection that runs retries just so can check how long it takes and that retrying is happening
[
https://issues.apache.org/jira/browse/HBASE-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302771#comment-17302771
]
Hudson commented on HBASE-25561:
Results for branch branch-2
[build #201 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Added ignored test for async connection that runs retries just so can check
> how long it takes and that retrying is happening
>
>
> Key: HBASE-25561
> URL: https://issues.apache.org/jira/browse/HBASE-25561
> Project: HBase
> Issue Type: Test
> Components: test
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Trivial
> Fix For: 2.5.0
>
>
> There's an old @Ignored test in TestClientNoCluster that has been useful from
> time to time figuring how long N retries takes and reproducing certain
> production 'Connection rejected', etc., stack traces. This issue adds an
> @Ignored method for the async connection case so can do similar for async (I
> needed it just now because it looked like retries at a certain juncture were
> not happening -- turns out they are... this addition helped me prove this the
> case).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25595) TestLruBlockCache.testBackgroundEvictionThread is flaky
[
https://issues.apache.org/jira/browse/HBASE-25595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302772#comment-17302772
]
Hudson commented on HBASE-25595:
Results for branch branch-2
[build #201 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/201/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TestLruBlockCache.testBackgroundEvictionThread is flaky
> ---
>
> Key: HBASE-25595
> URL: https://issues.apache.org/jira/browse/HBASE-25595
> Project: HBase
> Issue Type: Bug
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.3
>
>
> As the test result in
> [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2700/2/testReport/org.apache.hadoop.hbase.io.hfile/TestLruBlockCache/precommit_checks___yetus_jdk11_hadoop3_checks___testBackgroundEvictionThread/]
> It fails because background EvictionThread used the weak reference of
> LRUBlockCache, when the cache object is cleared by garbage collector, NO
> block eviction will be triggered by LruBlockCache#runEviction().
> .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] saintstack commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
saintstack commented on pull request #3001: URL: https://github.com/apache/hbase/pull/3001#issuecomment-800481912 Shout if you need input on how to fix the licensing issue. The test failure looks related. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #3001: HBASE-25518 Support separate child regions to different region servers
saintstack commented on a change in pull request #3001:
URL: https://github.com/apache/hbase/pull/3001#discussion_r595411259
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionSplit.java
##
@@ -146,8 +152,70 @@ public void testSplitTableRegion() throws Exception {
UTIL.getAdmin().enableTable(tableName);
Thread.sleep(500);
-assertEquals("Table region not correct.", 2,
-UTIL.getHBaseCluster().getRegions(tableName).size());
+List tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+assertEquals("Table region not correct.", 2, tableRegions.size());
+Map regionInfoMap =
UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+assertEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
+ regionInfoMap.get(tableRegions.get(1).getRegionInfo()));
+ }
+
+ @Test
+ public void testSplitTableRegionAndSeparateChildRegions() throws Exception {
+cleanupTest();
+//restart master
+
UTIL.getConfiguration().setBoolean(HConstants.HBASE_ENABLE_SEPARATE_CHILD_REGIONS,
true);
+setupCluster();
+
+final TableName tableName = TableName.valueOf(name.getMethodName());
+final ProcedureExecutor procExec =
getMasterProcedureExecutor();
+
+RegionInfo[] regions =
+ MasterProcedureTestingUtility.createTable(procExec, tableName, null,
ColumnFamilyName);
+insertData(tableName);
+int splitRowNum = startRowNum + rowCount / 2;
+byte[] splitKey = Bytes.toBytes("" + splitRowNum);
+
+assertTrue("not able to find a splittable region", regions != null);
+assertTrue("not able to find a splittable region", regions.length == 1);
+
+// Split region of the table
+long procId = procExec.submitProcedure(
+ new SplitTableRegionProcedure(procExec.getEnvironment(), regions[0],
splitKey));
+// Wait the completion
+ProcedureTestingUtility.waitProcedure(procExec, procId);
+ProcedureTestingUtility.assertProcNotFailed(procExec, procId);
+
+assertTrue("not able to split table",
UTIL.getHBaseCluster().getRegions(tableName).size() == 2);
+
+//disable table
+UTIL.getAdmin().disableTable(tableName);
+Thread.sleep(500);
+
+//stop master
+UTIL.getHBaseCluster().stopMaster(0);
+UTIL.getHBaseCluster().waitOnMaster(0);
+Thread.sleep(500);
+
+//restart master
+JVMClusterUtil.MasterThread t = UTIL.getHBaseCluster().startMaster();
+Thread.sleep(500);
+
+UTIL.invalidateConnection();
+// enable table
+UTIL.getAdmin().enableTable(tableName);
+Thread.sleep(500);
+
+List tableRegions = UTIL.getHBaseCluster().getRegions(tableName);
+assertEquals("Table region not correct.", 2, tableRegions.size());
+Map regionInfoMap =
UTIL.getHBaseCluster().getMaster().getAssignmentManager()
+ .getRegionStates().getRegionAssignments();
+assertNotEquals(regionInfoMap.get(tableRegions.get(0).getRegionInfo()),
Review comment:
+1
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Apache-HBase commented on pull request #3001: URL: https://github.com/apache/hbase/pull/3001#issuecomment-800466111 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 40s | master passed | | +1 :green_heart: | compile | 1m 38s | master passed | | +1 :green_heart: | shadedjars | 7m 30s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 43s | the patch passed | | +1 :green_heart: | compile | 1m 40s | the patch passed | | +1 :green_heart: | javac | 1m 40s | the patch passed | | -1 :x: | shadedjars | 5m 58s | patch has 10 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 48s | hbase-common in the patch passed. | | -1 :x: | unit | 212m 37s | hbase-server in the patch failed. | | | | 246m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3001 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c0c01a7502bf 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | Default Java | AdoptOpenJDK-11.0.10+9 | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/testReport/ | | Max. process+thread count | 3546 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Apache-HBase commented on pull request #3001: URL: https://github.com/apache/hbase/pull/3001#issuecomment-800462769 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 2s | master passed | | +1 :green_heart: | compile | 1m 22s | master passed | | +1 :green_heart: | shadedjars | 7m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 57s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 52s | the patch passed | | +1 :green_heart: | compile | 1m 22s | the patch passed | | +1 :green_heart: | javac | 1m 22s | the patch passed | | -1 :x: | shadedjars | 5m 44s | patch has 10 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 33s | hbase-common in the patch passed. | | -1 :x: | unit | 211m 45s | hbase-server in the patch failed. | | | | 242m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3001 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 912b2ddb45b0 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/testReport/ | | Max. process+thread count | 3902 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25595) TestLruBlockCache.testBackgroundEvictionThread is flaky
[
https://issues.apache.org/jira/browse/HBASE-25595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302720#comment-17302720
]
Hudson commented on HBASE-25595:
Results for branch branch-2.3
[build #186 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TestLruBlockCache.testBackgroundEvictionThread is flaky
> ---
>
> Key: HBASE-25595
> URL: https://issues.apache.org/jira/browse/HBASE-25595
> Project: HBase
> Issue Type: Bug
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.3
>
>
> As the test result in
> [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2700/2/testReport/org.apache.hadoop.hbase.io.hfile/TestLruBlockCache/precommit_checks___yetus_jdk11_hadoop3_checks___testBackgroundEvictionThread/]
> It fails because background EvictionThread used the weak reference of
> LRUBlockCache, when the cache object is cleared by garbage collector, NO
> block eviction will be triggered by LruBlockCache#runEviction().
> .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25597) Add row info in Exception when cell size exceeds maxCellSize
[
https://issues.apache.org/jira/browse/HBASE-25597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302722#comment-17302722
]
Hudson commented on HBASE-25597:
Results for branch branch-2.3
[build #186 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add row info in Exception when cell size exceeds maxCellSize
>
>
> Key: HBASE-25597
> URL: https://issues.apache.org/jira/browse/HBASE-25597
> Project: HBase
> Issue Type: Improvement
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3
>
>
> When cell size exceeds maxCellSize(default is 10M), client will get a
> DoNotRetryIOException, code as below:
> {code:java}
> private void checkCellSizeLimit(final HRegion r, final Mutation m) throws
> IOException {
> if (r.maxCellSize > 0) {
> CellScanner cells = m.cellScanner();
> while (cells.advance()) {
> int size = PrivateCellUtil.estimatedSerializedSizeOf(cells.current());
> if (size > r.maxCellSize) {
> String msg = "Cell with size " + size + " exceeds limit of " +
> r.maxCellSize + " bytes";
> LOG.debug(msg);
> throw new DoNotRetryIOException(msg);
> }
> }
> }
> }
> {code}
> There is no row related information, which makes troubleshooting difficult.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25374) Make REST Client connection and socket time out configurable
[
https://issues.apache.org/jira/browse/HBASE-25374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302723#comment-17302723
]
Hudson commented on HBASE-25374:
Results for branch branch-2.3
[build #186 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Make REST Client connection and socket time out configurable
>
>
> Key: HBASE-25374
> URL: https://issues.apache.org/jira/browse/HBASE-25374
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Affects Versions: 3.0.0-alpha-1, 2.3.4, 2.5.0, 2.4.3
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3
>
>
> Currently there is no way to configure connection and socket timeout for REST
> client,
> https://github.com/apache/hbase/blob/7d0a687e5798a2f4ca3190b409169f7e17a75b34/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/Client.java#L102
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25621) Balancer should check region plan source to avoid misplace region groups
[
https://issues.apache.org/jira/browse/HBASE-25621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302721#comment-17302721
]
Hudson commented on HBASE-25621:
Results for branch branch-2.3
[build #186 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Balancer should check region plan source to avoid misplace region groups
>
>
> Key: HBASE-25621
> URL: https://issues.apache.org/jira/browse/HBASE-25621
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.3
>
>
> Currently, balancer generates all group plans firstly, and then execute
> overall plans. When there are table regions movement between groups but after
> the groups plans generating and before executing, balancer may misplace the
> region to the old group. But this is not serious problem, next balancer will
> fix it.
> If we add a check before executing balance plans, we can avoid the problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25635) CandidateGenerator may miss some region balance actions
[
https://issues.apache.org/jira/browse/HBASE-25635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302718#comment-17302718
]
Hudson commented on HBASE-25635:
Results for branch branch-2.3
[build #186 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> CandidateGenerator may miss some region balance actions
> ---
>
> Key: HBASE-25635
> URL: https://issues.apache.org/jira/browse/HBASE-25635
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Affects Versions: 2.0.0
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3
>
> Attachments: 1614940188041-image.png
>
>
> {color:#172b4d}In codes of CandidateGenerator#getAction()，valid region index
> are greater or equal to zero. Currently only regions with greater than zero
> indexes can get balance actions.{color}
> {code:java}
> protected BaseLoadBalancer.Cluster.Action getAction(int fromServer, int
> fromRegion,
> int toServer, int toRegion) {
> if (fromServer < 0 || toServer < 0) {
> return BaseLoadBalancer.Cluster.NullAction;
> }
> if (fromRegion > 0 && toRegion > 0) {
> return new BaseLoadBalancer.Cluster.SwapRegionsAction(fromServer,
> fromRegion,
> toServer, toRegion);
> } else if (fromRegion > 0) {
> return new BaseLoadBalancer.Cluster.MoveRegionAction(fromRegion,
> fromServer, toServer);
> } else if (toRegion > 0) {
> return new BaseLoadBalancer.Cluster.MoveRegionAction(toRegion, toServer,
> fromServer);
> } else {
> return BaseLoadBalancer.Cluster.NullAction;
> }
> }{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25662) Fix spotbugs warning in RoundRobinTableInputFormat
[
https://issues.apache.org/jira/browse/HBASE-25662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302719#comment-17302719
]
Hudson commented on HBASE-25662:
Results for branch branch-2.3
[build #186 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/186/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix spotbugs warning in RoundRobinTableInputFormat
> --
>
> Key: HBASE-25662
> URL: https://issues.apache.org/jira/browse/HBASE-25662
> Project: HBase
> Issue Type: Bug
> Components: findbugs
>Reporter: Pankaj Kumar
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3
>
>
> RoundRobinTableInputFormat.roundRobin(List) makes inefficient use of keySet
> iterator instead of entrySet iterator,
> https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3043/2/artifact/yetus-general-check/output/branch-spotbugs-root-warnings.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] wchevreuil opened a new pull request #3055: HBASE-25656 Backport to branch-2.2: [HBASE-25548 Optionally allow sna…
wchevreuil opened a new pull request #3055: URL: https://github.com/apache/hbase/pull/3055 …pshots to preserve cluster's max filesize config by setting it into table descriptor] This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25621) Balancer should check region plan source to avoid misplace region groups
[ https://issues.apache.org/jira/browse/HBASE-25621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-25621. --- Fix Version/s: 2.2.7 Resolution: Fixed Pushed to branch-2.2. > Balancer should check region plan source to avoid misplace region groups > > > Key: HBASE-25621 > URL: https://issues.apache.org/jira/browse/HBASE-25621 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.3 > > > Currently, balancer generates all group plans firstly, and then execute > overall plans. When there are table regions movement between groups but after > the groups plans generating and before executing, balancer may misplace the > region to the old group. But this is not serious problem, next balancer will > fix it. > If we add a check before executing balance plans, we can avoid the problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache-HBase commented on pull request #1708: URL: https://github.com/apache/hbase/pull/1708#issuecomment-800421763 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 45s | master passed | | +1 :green_heart: | compile | 2m 45s | master passed | | +1 :green_heart: | shadedjars | 7m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 10m 56s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 47s | the patch passed | | +1 :green_heart: | compile | 2m 43s | the patch passed | | +1 :green_heart: | javac | 2m 43s | the patch passed | | +1 :green_heart: | shadedjars | 7m 51s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 11m 3s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 300m 33s | root in the patch failed. | | | | 358m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/11/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1708 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0d5aaba55962 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 59ec375d67 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/11/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/11/testReport/ | | Max. process+thread count | 5182 (vs. ulimit of 3) | | modules | C: hbase-logging hbase-common hbase-metrics-api hbase-metrics hbase-hadoop-compat hbase-client hbase-zookeeper hbase-replication hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-testing-util hbase-thrift hbase-shell hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-shaded hbase-shaded/hbase-shaded-client hbase-shaded/hbase-shaded-client-byo-hadoop hbase-shaded/hbase-shaded-mapreduce hbase-hbtop hbase-assembly hbase-shaded/hbase-shaded-testing-util hbase-shaded/hbase-shaded-testing-util-tester hbase-shaded/hbase-shaded-check-invariants hbase-shaded/hbase-shaded-with-hadoop-check-invariants hbase-archetypes/hbase-client-project hbase-archetypes/hbase-shaded-client-project . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/11/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (HBASE-25621) Balancer should check region plan source to avoid misplace region groups
[ https://issues.apache.org/jira/browse/HBASE-25621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-25621: --- > Balancer should check region plan source to avoid misplace region groups > > > Key: HBASE-25621 > URL: https://issues.apache.org/jira/browse/HBASE-25621 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3 > > > Currently, balancer generates all group plans firstly, and then execute > overall plans. When there are table regions movement between groups but after > the groups plans generating and before executing, balancer may misplace the > region to the old group. But this is not serious problem, next balancer will > fix it. > If we add a check before executing balance plans, we can avoid the problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25595) TestLruBlockCache.testBackgroundEvictionThread is flaky
[ https://issues.apache.org/jira/browse/HBASE-25595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-25595. --- Fix Version/s: 2.2.7 Resolution: Fixed > TestLruBlockCache.testBackgroundEvictionThread is flaky > --- > > Key: HBASE-25595 > URL: https://issues.apache.org/jira/browse/HBASE-25595 > Project: HBase > Issue Type: Bug >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.5.0, 2.3.5, 2.4.3 > > > As the test result in > [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2700/2/testReport/org.apache.hadoop.hbase.io.hfile/TestLruBlockCache/precommit_checks___yetus_jdk11_hadoop3_checks___testBackgroundEvictionThread/] > It fails because background EvictionThread used the weak reference of > LRUBlockCache, when the cache object is cleared by garbage collector, NO > block eviction will be triggered by LruBlockCache#runEviction(). > . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-25595) TestLruBlockCache.testBackgroundEvictionThread is flaky
[ https://issues.apache.org/jira/browse/HBASE-25595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reopened HBASE-25595: --- Reopening for branch-2.2 backport. > TestLruBlockCache.testBackgroundEvictionThread is flaky > --- > > Key: HBASE-25595 > URL: https://issues.apache.org/jira/browse/HBASE-25595 > Project: HBase > Issue Type: Bug >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0, 2.3.5, 2.4.3 > > > As the test result in > [https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2700/2/testReport/org.apache.hadoop.hbase.io.hfile/TestLruBlockCache/precommit_checks___yetus_jdk11_hadoop3_checks___testBackgroundEvictionThread/] > It fails because background EvictionThread used the weak reference of > LRUBlockCache, when the cache object is cleared by garbage collector, NO > block eviction will be triggered by LruBlockCache#runEviction(). > . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24528) Improve balancer decision observability
[ https://issues.apache.org/jira/browse/HBASE-24528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302576#comment-17302576 ] Michael Stack commented on HBASE-24528: --- [~vjasani] Thanks. > Improve balancer decision observability > --- > > Key: HBASE-24528 > URL: https://issues.apache.org/jira/browse/HBASE-24528 > Project: HBase > Issue Type: New Feature > Components: Admin, Balancer, Operability, shell, UI >Reporter: Andrew Kyle Purtell >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0 > > Attachments: Screenshot 2020-08-12 at 11.50.43 PM.png, Screenshot > 2020-08-15 at 8.08.57 PM.png > > > We provide detailed INFO and DEBUG level logging of balancer decision > factors, outcome, and reassignment planning, as well as similarly detailed > logging of the resulting assignment manager activity. However, an operator > may need to perform online and interactive observation, debugging, or > performance analysis of current balancer activity. Scraping and correlating > the many log lines resulting from a balancer execution is labor intensive and > has a lot of latency (order of ~minutes to acquire and index, order of > ~minutes to correlate). > The balancer should maintain a rolling window of history, e.g. the last 100 > region move plans, or last 1000 region move plans submitted to the assignment > manager. This history should include decision factor details and weights and > costs. The rsgroups balancer may be able to provide fairly simple decision > factors, like for example "this table was reassigned to that regionserver > group". The underlying or vanilla stochastic balancer on the other hand, > after a walk over random assignment plans, will have considered a number of > cost functions with various inputs (locality, load, etc.) and multipliers, > including custom cost functions. We can devise an extensible class structure > that represents explanations for balancer decisions, and for each region move > plan that is actually submitted to the assignment manager, we can keep the > explanations of all relevant decision factors alongside the other details of > the assignment plan like the region name, and the source and destination > regionservers. > This history should be available via API for use by new shell commands and > admin UI widgets. > The new shell commands and UI widgets can unpack the representation of > balancer decision components into human readable output. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3001: HBASE-25518 Support separate child regions to different region servers
Apache-HBase commented on pull request #3001: URL: https://github.com/apache/hbase/pull/3001#issuecomment-800299322 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 48s | master passed | | +1 :green_heart: | compile | 4m 3s | master passed | | +1 :green_heart: | checkstyle | 1m 28s | master passed | | +1 :green_heart: | spotbugs | 2m 51s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 3m 59s | the patch passed | | +1 :green_heart: | javac | 3m 59s | the patch passed | | -0 :warning: | checkstyle | 1m 3s | hbase-server: The patch generated 2 new + 1 unchanged - 1 fixed = 3 total (was 2) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 33s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 3m 11s | the patch passed | ||| _ Other Tests _ | | -1 :x: | asflicense | 0m 27s | The patch generated 1 ASF License warnings. | | | | 52m 13s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3001 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux b9a1b3e0e0a8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / d74ae15fa7 | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | asflicense | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/artifact/yetus-general-check/output/patch-asflicense-problems.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3001/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1708: HBASE-19577 Use log4j2 instead of log4j for logging
Apache-HBase commented on pull request #1708: URL: https://github.com/apache/hbase/pull/1708#issuecomment-800290624 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 1s | master passed | | +1 :green_heart: | compile | 2m 50s | master passed | | +1 :green_heart: | shadedjars | 6m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 12m 57s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 8s | the patch passed | | +1 :green_heart: | compile | 2m 58s | the patch passed | | +1 :green_heart: | javac | 2m 58s | the patch passed | | +1 :green_heart: | shadedjars | 7m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 13m 4s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 149m 41s | root in the patch failed. | | | | 213m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/11/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1708 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 46f3aba6b6eb 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 59ec375d67 | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/11/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/11/testReport/ | | Max. process+thread count | 4122 (vs. ulimit of 3) | | modules | C: hbase-logging hbase-common hbase-metrics-api hbase-metrics hbase-hadoop-compat hbase-client hbase-zookeeper hbase-replication hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-testing-util hbase-thrift hbase-shell hbase-endpoint hbase-backup hbase-it hbase-rest hbase-examples hbase-shaded hbase-shaded/hbase-shaded-client hbase-shaded/hbase-shaded-client-byo-hadoop hbase-shaded/hbase-shaded-mapreduce hbase-hbtop hbase-assembly hbase-shaded/hbase-shaded-testing-util hbase-shaded/hbase-shaded-testing-util-tester hbase-shaded/hbase-shaded-check-invariants hbase-shaded/hbase-shaded-with-hadoop-check-invariants hbase-archetypes/hbase-client-project hbase-archetypes/hbase-shaded-client-project . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-1708/11/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
