Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
wchevreuil merged PR #5793: URL: https://github.com/apache/hbase/pull/5793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
vinayakphegde commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2043004482 @wchevreuil, it seems like most of the tests have passed, and any failures were due to flaky tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
Apache-HBase commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2042968954 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 27s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 44s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 28s | the patch passed | | +1 :green_heart: | compile | 0m 43s | the patch passed | | +1 :green_heart: | javac | 0m 43s | the patch passed | | +1 :green_heart: | shadedjars | 5m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 278m 11s | hbase-server in the patch failed. | | | | 300m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5793 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 30591c443986 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / 28c1e3b2a6 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/4/testReport/ | | Max. process+thread count | 5764 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
Apache-HBase commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2042894870 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 56s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 55s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 49s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 50s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | +1 :green_heart: | shadedjars | 6m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 248m 11s | hbase-server in the patch passed. | | | | 273m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5793 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 949ec8740130 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / 28c1e3b2a6 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/4/testReport/ | | Max. process+thread count | 4817 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
jhungund commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1555856302
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The DataTieringManager class categorizes data into hot data and cold data
based on the specified
+ * {@link DataTieringType} when DataTiering is enabled. DataTiering is
disabled by default with
+ * {@link DataTieringType} set to {@link DataTieringType#NONE}. The {@link
DataTieringType}
+ * determines the logic for distinguishing data into hot or cold. By default,
all data is considered
+ * as hot.
+ */
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ /**
+ * Initializes the DataTieringManager instance with the provided map of
online regions.
+ * @param onlineRegions A map containing online regions.
+ */
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ /**
+ * Retrieves the instance of DataTieringManager.
+ * @return The instance of DataTieringManager.
+ * @throws IllegalStateException if DataTieringManager has not been
instantiated.
+ */
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ /**
+ * Determines whether data tiering is enabled for the given block cache key.
+ * @param key the block cache key
+ * @return {@code true} if data tiering is enabled for the HFile associated
with the key,
+ * {@code false} otherwise
+ * @throws DataTieringException if there is an error retrieving the HFile
path or configuration
+ */
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ /**
+ * Determines whether data tiering is enabled for the given HFile path.
+ * @param hFilePath the path to the HFile
+ * @return {@code true} if data tiering is enabled, {@code false} otherwise
+ * @throws DataTieringException if there is an error retrieving the
configuration
+ */
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+return !dataTieringType.equals(DataTieringType.NONE);
+ }
+
+ /**
+ * Determines whether the data associated with the given block cache key is
considered hot.
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
vinayakphegde commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1555847930
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The DataTieringManager class categorizes data into hot data and cold data
based on the specified
+ * {@link DataTieringType} when DataTiering is enabled. DataTiering is
disabled by default with
+ * {@link DataTieringType} set to {@link DataTieringType#NONE}. The {@link
DataTieringType}
+ * determines the logic for distinguishing data into hot or cold. By default,
all data is considered
+ * as hot.
+ */
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ /**
+ * Initializes the DataTieringManager instance with the provided map of
online regions.
+ * @param onlineRegions A map containing online regions.
+ */
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ /**
+ * Retrieves the instance of DataTieringManager.
+ * @return The instance of DataTieringManager.
+ * @throws IllegalStateException if DataTieringManager has not been
instantiated.
+ */
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ /**
+ * Determines whether data tiering is enabled for the given block cache key.
+ * @param key the block cache key
+ * @return {@code true} if data tiering is enabled for the HFile associated
with the key,
+ * {@code false} otherwise
+ * @throws DataTieringException if there is an error retrieving the HFile
path or configuration
+ */
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ /**
+ * Determines whether data tiering is enabled for the given HFile path.
+ * @param hFilePath the path to the HFile
+ * @return {@code true} if data tiering is enabled, {@code false} otherwise
+ * @throws DataTieringException if there is an error retrieving the
configuration
+ */
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+return !dataTieringType.equals(DataTieringType.NONE);
+ }
+
+ /**
+ * Determines whether the data associated with the given block cache key is
considered
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
jhungund commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1555814598
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The DataTieringManager class categorizes data into hot data and cold data
based on the specified
+ * {@link DataTieringType} when DataTiering is enabled. DataTiering is
disabled by default with
+ * {@link DataTieringType} set to {@link DataTieringType#NONE}. The {@link
DataTieringType}
+ * determines the logic for distinguishing data into hot or cold. By default,
all data is considered
+ * as hot.
+ */
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ /**
+ * Initializes the DataTieringManager instance with the provided map of
online regions.
+ * @param onlineRegions A map containing online regions.
+ */
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ /**
+ * Retrieves the instance of DataTieringManager.
+ * @return The instance of DataTieringManager.
+ * @throws IllegalStateException if DataTieringManager has not been
instantiated.
+ */
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ /**
+ * Determines whether data tiering is enabled for the given block cache key.
+ * @param key the block cache key
+ * @return {@code true} if data tiering is enabled for the HFile associated
with the key,
+ * {@code false} otherwise
+ * @throws DataTieringException if there is an error retrieving the HFile
path or configuration
+ */
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ /**
+ * Determines whether data tiering is enabled for the given HFile path.
+ * @param hFilePath the path to the HFile
+ * @return {@code true} if data tiering is enabled, {@code false} otherwise
+ * @throws DataTieringException if there is an error retrieving the
configuration
+ */
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+return !dataTieringType.equals(DataTieringType.NONE);
+ }
+
+ /**
+ * Determines whether the data associated with the given block cache key is
considered hot.
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
Apache-HBase commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2042385473 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | HBASE-28463 passed | | +1 :green_heart: | compile | 2m 38s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 37s | HBASE-28463 passed | | +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 37s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 58s | the patch passed | | +1 :green_heart: | compile | 2m 39s | the patch passed | | +1 :green_heart: | javac | 2m 39s | the patch passed | | -0 :warning: | checkstyle | 0m 37s | hbase-server: The patch generated 8 new + 4 unchanged - 0 fixed = 12 total (was 4) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 40s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 43s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 31m 17s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5793 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux e517e1c5bde6 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / 28c1e3b2a6 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 81 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/4/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
vinayakphegde commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2042266090 > LTGM, can we just address latest spotless failure? That's because of the Javadoc in the `TestDataTieringManager` class, where I included the structure of the `TestDataTieringManager#hStoreFiles` for better code comprehension. What do you think we should do instead? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
wchevreuil commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2042252076 LTGM, can we just address latest spotless failure? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
Apache-HBase commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2040824093 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 43s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 12s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 30s | the patch passed | | +1 :green_heart: | compile | 0m 44s | the patch passed | | +1 :green_heart: | javac | 0m 44s | the patch passed | | +1 :green_heart: | shadedjars | 5m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 248m 38s | hbase-server in the patch passed. | | | | 271m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5793 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ef1ff8a4a597 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / 28c1e3b2a6 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/3/testReport/ | | Max. process+thread count | 5053 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
Apache-HBase commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2040820860 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 52s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 51s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 47s | the patch passed | | +1 :green_heart: | compile | 0m 49s | the patch passed | | +1 :green_heart: | javac | 0m 49s | the patch passed | | +1 :green_heart: | shadedjars | 5m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 240m 27s | hbase-server in the patch passed. | | | | 263m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5793 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a8b35c1b787d 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / 28c1e3b2a6 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/3/testReport/ | | Max. process+thread count | 4671 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/3/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
Apache-HBase commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2040590074 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | HBASE-28463 passed | | +1 :green_heart: | compile | 2m 48s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 38s | HBASE-28463 passed | | +1 :green_heart: | spotless | 0m 49s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 43s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 18s | the patch passed | | +1 :green_heart: | compile | 2m 44s | the patch passed | | +1 :green_heart: | javac | 2m 44s | the patch passed | | -0 :warning: | checkstyle | 0m 37s | hbase-server: The patch generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 6m 48s | Patch does not cause any errors with Hadoop 3.3.6. | | -1 :x: | spotless | 0m 50s | patch has 46 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 2m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. | | | | 35m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/3/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5793 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 7f8b257c1b65 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / 28c1e3b2a6 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/3/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 81 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/3/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
Apache-HBase commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2040537932 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 19s | HBASE-28463 passed | | +1 :green_heart: | compile | 1m 1s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 6m 27s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 28s | the patch passed | | +1 :green_heart: | compile | 1m 14s | the patch passed | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | shadedjars | 6m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 292m 22s | hbase-server in the patch failed. | | | | 321m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5793 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0331e5823b27 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / 28c1e3b2a6 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/2/testReport/ | | Max. process+thread count | 4988 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
Apache-HBase commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2040481765 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 2s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 50s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 47s | the patch passed | | +1 :green_heart: | compile | 0m 52s | the patch passed | | +1 :green_heart: | javac | 0m 52s | the patch passed | | +1 :green_heart: | shadedjars | 5m 21s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 251m 27s | hbase-server in the patch failed. | | | | 275m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5793 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e0f2e8772541 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / 28c1e3b2a6 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/2/testReport/ | | Max. process+thread count | 4710 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
wchevreuil commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1553980810
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -0,0 +1,388 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.fs.HFileSystem;
+import org.apache.hadoop.hbase.io.hfile.BlockCache;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheFactory;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.io.hfile.BlockType;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.testclassification.SmallTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+/**
+ * This class is used to test the functionality of the DataTieringManager.
+ *
+ * The mock online regions are stored in {@link
TestDataTieringManager#testOnlineRegions}.
+ * For all tests, the setup of {@link
TestDataTieringManager#testOnlineRegions} occurs only once.
+ * Please refer to {@link TestDataTieringManager#setupOnlineRegions()} for the
structure.
+ * Additionally, a list of all store files is maintained in {@link
TestDataTieringManager#hStoreFiles}.
+ * The characteristics of these store files are listed below:
+ * ## HStoreFile Information
+ *
+ * | HStoreFile | Region | Store | DataTiering
| isHot |
+ *
|--||-|---|---|
+ * | hStoreFile0 | region1| hStore11| TIME_RANGE
| true |
+ * | hStoreFile1 | region1| hStore12| NONE
| true |
+ * | hStoreFile2 | region2| hStore21| TIME_RANGE
| true |
+ * | hStoreFile3 | region2| hStore22| TIME_RANGE
| false |
+ */
+
+@Category({ RegionServerTests.class, SmallTests.class })
+public class TestDataTieringManager {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestDataTieringManager.class);
+
+ private static final HBaseTestingUtil TEST_UTIL = new HBaseTestingUtil();
+ private static Configuration defaultConf;
+ private static FileSystem fs;
+ private static CacheConfig cacheConf;
+ private static Path testDir;
+ private static Map testOnlineRegions;
+
+ private static DataTieringManager dataTieringManager;
+ private static List hStoreFiles;
+
+ @BeforeClass
+ public static void setupBeforeClass() throws Exception {
+testDir =
TEST_UTIL.getDataTestDir(TestDataTieringManager.class.getSimpleName());
+defaultConf = TEST_UTIL.getConfiguration();
+fs = HFileSystem.get(defaultConf);
+BlockCache blockCache = BlockCacheFactory.createBlockCache(defaultConf);
+cacheConf = new CacheConfig(defaultConf, blockCache);
+
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
wchevreuil commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1553979182
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -0,0 +1,388 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.fs.HFileSystem;
+import org.apache.hadoop.hbase.io.hfile.BlockCache;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheFactory;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.io.hfile.BlockType;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.testclassification.SmallTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+/**
+ * This class is used to test the functionality of the DataTieringManager.
+ *
+ * The mock online regions are stored in {@link
TestDataTieringManager#testOnlineRegions}.
+ * For all tests, the setup of {@link
TestDataTieringManager#testOnlineRegions} occurs only once.
+ * Please refer to {@link TestDataTieringManager#setupOnlineRegions()} for the
structure.
+ * Additionally, a list of all store files is maintained in {@link
TestDataTieringManager#hStoreFiles}.
+ * The characteristics of these store files are listed below:
+ * ## HStoreFile Information
+ *
+ * | HStoreFile | Region | Store | DataTiering
| isHot |
+ *
|--||-|---|---|
+ * | hStoreFile0 | region1| hStore11| TIME_RANGE
| true |
+ * | hStoreFile1 | region1| hStore12| NONE
| true |
+ * | hStoreFile2 | region2| hStore21| TIME_RANGE
| true |
+ * | hStoreFile3 | region2| hStore22| TIME_RANGE
| false |
+ */
+
+@Category({ RegionServerTests.class, SmallTests.class })
+public class TestDataTieringManager {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestDataTieringManager.class);
+
+ private static final HBaseTestingUtil TEST_UTIL = new HBaseTestingUtil();
+ private static Configuration defaultConf;
+ private static FileSystem fs;
+ private static CacheConfig cacheConf;
+ private static Path testDir;
+ private static Map testOnlineRegions;
+
+ private static DataTieringManager dataTieringManager;
+ private static List hStoreFiles;
+
+ @BeforeClass
+ public static void setupBeforeClass() throws Exception {
+testDir =
TEST_UTIL.getDataTestDir(TestDataTieringManager.class.getSimpleName());
+defaultConf = TEST_UTIL.getConfiguration();
+fs = HFileSystem.get(defaultConf);
+BlockCache blockCache = BlockCacheFactory.createBlockCache(defaultConf);
+cacheConf = new CacheConfig(defaultConf, blockCache);
+
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
Apache-HBase commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2040058881 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 54s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 16s | HBASE-28463 passed | | +1 :green_heart: | compile | 3m 16s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 43s | HBASE-28463 passed | | +1 :green_heart: | spotless | 0m 58s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 5s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | compile | 2m 58s | the patch passed | | +1 :green_heart: | javac | 2m 58s | the patch passed | | -0 :warning: | checkstyle | 0m 45s | hbase-server: The patch generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4) | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 6m 32s | Patch does not cause any errors with Hadoop 3.3.6. | | -1 :x: | spotless | 0m 46s | patch has 46 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 2m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 37m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5793 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 7c12c5b4ff03 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / 28c1e3b2a6 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/2/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/2/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
vinayakphegde commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1553348860
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+return !dataTieringType.equals(DataTieringType.NONE);
+ }
+
+ public boolean isHotData(BlockCacheKey key) throws DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isHotData(hFilePath);
+ }
+
+ public boolean isHotData(Path hFilePath) throws DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+
+if (dataTieringType.equals(DataTieringType.TIME_RANGE)) {
+ long hotDataAge = getDataTieringHotDataAge(configuration);
+
+ HStoreFile hStoreFile = getHStoreFile(hFilePath);
+ if (hStoreFile == null) {
+throw new DataTieringException(
+ "HStoreFile corresponding to " + hFilePath + " doesn't exist");
+ }
+ OptionalLong maxTimestamp = hStoreFile.getMaximumTimestamp();
+ if (!maxTimestamp.isPresent()) {
+throw new DataTieringException("Maximum timestamp not present for " +
hFilePath);
+ }
+
+ long currentTimestamp =
EnvironmentEdgeManager.getDelegate().currentTime();
+ long diff = currentTimestamp - maxTimestamp.getAsLong();
+ return diff <= hotDataAge;
+}
+return false;
Review Comment:
Okay, I get your point. then we can eliminate isDataTieringEnabled methods,
right? since we are considering everything as hot by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
vinayakphegde commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1553318202
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+return !dataTieringType.equals(DataTieringType.NONE);
+ }
+
+ public boolean isHotData(BlockCacheKey key) throws DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isHotData(hFilePath);
+ }
+
+ public boolean isHotData(Path hFilePath) throws DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+
+if (dataTieringType.equals(DataTieringType.TIME_RANGE)) {
+ long hotDataAge = getDataTieringHotDataAge(configuration);
+
+ HStoreFile hStoreFile = getHStoreFile(hFilePath);
+ if (hStoreFile == null) {
+throw new DataTieringException(
+ "HStoreFile corresponding to " + hFilePath + " doesn't exist");
+ }
+ OptionalLong maxTimestamp = hStoreFile.getMaximumTimestamp();
+ if (!maxTimestamp.isPresent()) {
+throw new DataTieringException("Maximum timestamp not present for " +
hFilePath);
+ }
+
+ long currentTimestamp =
EnvironmentEdgeManager.getDelegate().currentTime();
+ long diff = currentTimestamp - maxTimestamp.getAsLong();
+ return diff <= hotDataAge;
+}
+return false;
Review Comment:
Even with this implementation, it will function the same. For example, in
eviction, we'll prioritize evicting the cold data files first, which are all
the files that satisfy the condition (isDataTieringEnabled(file) &&
!isHotData(file)). Once I identify these
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
vinayakphegde commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1553288683
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+return !dataTieringType.equals(DataTieringType.NONE);
+ }
+
+ public boolean isHotData(BlockCacheKey key) throws DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isHotData(hFilePath);
+ }
+
+ public boolean isHotData(Path hFilePath) throws DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+
+if (dataTieringType.equals(DataTieringType.TIME_RANGE)) {
+ long hotDataAge = getDataTieringHotDataAge(configuration);
+
+ HStoreFile hStoreFile = getHStoreFile(hFilePath);
+ if (hStoreFile == null) {
+throw new DataTieringException(
Review Comment:
Got it. should we change in other places as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
wchevreuil commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1553240987
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+return !dataTieringType.equals(DataTieringType.NONE);
+ }
+
+ public boolean isHotData(BlockCacheKey key) throws DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isHotData(hFilePath);
+ }
+
+ public boolean isHotData(Path hFilePath) throws DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+
+if (dataTieringType.equals(DataTieringType.TIME_RANGE)) {
+ long hotDataAge = getDataTieringHotDataAge(configuration);
+
+ HStoreFile hStoreFile = getHStoreFile(hFilePath);
+ if (hStoreFile == null) {
+throw new DataTieringException(
Review Comment:
I'm thinking in terms of responsibilities and cohesion. Who should know how
to classify a block as hot or cold for a file that is not in the list of
regions stores? Shouldn't this be done here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
wchevreuil commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1553235776
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+return !dataTieringType.equals(DataTieringType.NONE);
+ }
+
+ public boolean isHotData(BlockCacheKey key) throws DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isHotData(hFilePath);
+ }
+
+ public boolean isHotData(Path hFilePath) throws DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+
+if (dataTieringType.equals(DataTieringType.TIME_RANGE)) {
+ long hotDataAge = getDataTieringHotDataAge(configuration);
+
+ HStoreFile hStoreFile = getHStoreFile(hFilePath);
+ if (hStoreFile == null) {
+throw new DataTieringException(
+ "HStoreFile corresponding to " + hFilePath + " doesn't exist");
+ }
+ OptionalLong maxTimestamp = hStoreFile.getMaximumTimestamp();
+ if (!maxTimestamp.isPresent()) {
+throw new DataTieringException("Maximum timestamp not present for " +
hFilePath);
+ }
+
+ long currentTimestamp =
EnvironmentEdgeManager.getDelegate().currentTime();
+ long diff = currentTimestamp - maxTimestamp.getAsLong();
+ return diff <= hotDataAge;
+}
+return false;
Review Comment:
> For example, if I set it to true, it means we are indicating that data is
hot even when data tiering is disabled.
Ok, I think there's a misunderstanding of concepts. Today, without this
DataTiering thing, the behaviour is to cache everything and evict based
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
wchevreuil commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1553235776
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+return !dataTieringType.equals(DataTieringType.NONE);
+ }
+
+ public boolean isHotData(BlockCacheKey key) throws DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isHotData(hFilePath);
+ }
+
+ public boolean isHotData(Path hFilePath) throws DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+
+if (dataTieringType.equals(DataTieringType.TIME_RANGE)) {
+ long hotDataAge = getDataTieringHotDataAge(configuration);
+
+ HStoreFile hStoreFile = getHStoreFile(hFilePath);
+ if (hStoreFile == null) {
+throw new DataTieringException(
+ "HStoreFile corresponding to " + hFilePath + " doesn't exist");
+ }
+ OptionalLong maxTimestamp = hStoreFile.getMaximumTimestamp();
+ if (!maxTimestamp.isPresent()) {
+throw new DataTieringException("Maximum timestamp not present for " +
hFilePath);
+ }
+
+ long currentTimestamp =
EnvironmentEdgeManager.getDelegate().currentTime();
+ long diff = currentTimestamp - maxTimestamp.getAsLong();
+ return diff <= hotDataAge;
+}
+return false;
Review Comment:
> For example, if I set it to true, it means we are indicating that data is
hot even when data tiering is disabled.
Ok, I think there's a misunderstanding of concepts. Today, without this
DataTiering thing, the behaviour is to cache everything and evict based
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
Apache-HBase commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2037800563 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 4s | HBASE-28463 passed | | +1 :green_heart: | compile | 1m 2s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 6m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 31s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 7m 42s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 269m 1s | hbase-server in the patch failed. | | | | 299m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5793 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 13e6d25cc16e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / 28c1e3b2a6 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/1/testReport/ | | Max. process+thread count | 4865 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
Apache-HBase commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2037736939 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 39s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 44s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 6s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 27s | the patch passed | | +1 :green_heart: | compile | 0m 42s | the patch passed | | +1 :green_heart: | javac | 0m 42s | the patch passed | | +1 :green_heart: | shadedjars | 5m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 245m 25s | hbase-server in the patch passed. | | | | 268m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5793 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5ffc7c6e8b3f 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / 28c1e3b2a6 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/1/testReport/ | | Max. process+thread count | 4663 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/1/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
vinayakphegde commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1552061992
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+return !dataTieringType.equals(DataTieringType.NONE);
+ }
+
+ public boolean isHotData(BlockCacheKey key) throws DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isHotData(hFilePath);
+ }
+
+ public boolean isHotData(Path hFilePath) throws DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+
+if (dataTieringType.equals(DataTieringType.TIME_RANGE)) {
+ long hotDataAge = getDataTieringHotDataAge(configuration);
+
+ HStoreFile hStoreFile = getHStoreFile(hFilePath);
+ if (hStoreFile == null) {
+throw new DataTieringException(
Review Comment:
Wouldn't it be better to throw the exception and let the caller decide what
action to take based on the context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
vinayakphegde commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1552053165
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -0,0 +1,378 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.fs.HFileSystem;
+import org.apache.hadoop.hbase.io.hfile.BlockCache;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheFactory;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.io.hfile.BlockType;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.testclassification.SmallTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+/**
+ * This class is used to test the functionality of the DataTieringManager.
+ *
+ * The mock online regions are stored in {@link
TestDataTieringManager#testOnlineRegions}.
+ * For all tests, the setup of {@link
TestDataTieringManager#testOnlineRegions} occurs only once.
+ * Please refer to {@link TestDataTieringManager#setupOnlineRegions()} for the
structure.
+ * Additionally, a list of all store files is maintained in {@link
TestDataTieringManager#hStoreFiles}.
+ * The characteristics of these store files are listed below:
+ * ## HStoreFile Information
+ *
+ * | HStoreFile | Region | Store | DataTiering
| isHot |
+ *
|--||-|---|---|
+ * | hStoreFile0 | region1| hStore11| TIME_RANGE
| true |
+ * | hStoreFile1 | region1| hStore12| NONE
| false |
+ * | hStoreFile2 | region2| hStore21| TIME_RANGE
| true |
+ * | hStoreFile3 | region2| hStore22| TIME_RANGE
| false |
+ */
+
+@Category({ RegionServerTests.class, SmallTests.class })
+public class TestDataTieringManager {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestDataTieringManager.class);
+
+ private static final HBaseTestingUtil TEST_UTIL = new HBaseTestingUtil();
+ private static Configuration defaultConf;
+ private static FileSystem fs;
+ private static CacheConfig cacheConf;
+ private static Path testDir;
+ private static Map testOnlineRegions;
+
+ private static DataTieringManager dataTieringManager;
+ private static List hStoreFiles;
+
+ @BeforeClass
+ public static void setupBeforeClass() throws Exception {
+testDir =
TEST_UTIL.getDataTestDir(TestDataTieringManager.class.getSimpleName());
+defaultConf = TEST_UTIL.getConfiguration();
+fs = HFileSystem.get(defaultConf);
+BlockCache blockCache = BlockCacheFactory.createBlockCache(defaultConf);
+cacheConf = new CacheConfig(defaultConf, blockCache);
+
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
vinayakphegde commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1552050096
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+return !dataTieringType.equals(DataTieringType.NONE);
+ }
+
+ public boolean isHotData(BlockCacheKey key) throws DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isHotData(hFilePath);
+ }
+
+ public boolean isHotData(Path hFilePath) throws DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+
+if (dataTieringType.equals(DataTieringType.TIME_RANGE)) {
+ long hotDataAge = getDataTieringHotDataAge(configuration);
+
+ HStoreFile hStoreFile = getHStoreFile(hFilePath);
+ if (hStoreFile == null) {
+throw new DataTieringException(
+ "HStoreFile corresponding to " + hFilePath + " doesn't exist");
+ }
+ OptionalLong maxTimestamp = hStoreFile.getMaximumTimestamp();
+ if (!maxTimestamp.isPresent()) {
+throw new DataTieringException("Maximum timestamp not present for " +
hFilePath);
Review Comment:
Sure, will do that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
vinayakphegde commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1552047719
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+return !dataTieringType.equals(DataTieringType.NONE);
+ }
+
+ public boolean isHotData(BlockCacheKey key) throws DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isHotData(hFilePath);
+ }
+
+ public boolean isHotData(Path hFilePath) throws DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+
+if (dataTieringType.equals(DataTieringType.TIME_RANGE)) {
+ long hotDataAge = getDataTieringHotDataAge(configuration);
+
+ HStoreFile hStoreFile = getHStoreFile(hFilePath);
+ if (hStoreFile == null) {
+throw new DataTieringException(
+ "HStoreFile corresponding to " + hFilePath + " doesn't exist");
+ }
+ OptionalLong maxTimestamp = hStoreFile.getMaximumTimestamp();
+ if (!maxTimestamp.isPresent()) {
+throw new DataTieringException("Maximum timestamp not present for " +
hFilePath);
+ }
+
+ long currentTimestamp =
EnvironmentEdgeManager.getDelegate().currentTime();
+ long diff = currentTimestamp - maxTimestamp.getAsLong();
+ return diff <= hotDataAge;
+}
+return false;
Review Comment:
The logic reaches that point only when the `DataTieringType` is set to
`NONE`, indicating that data tiering is disabled. If data tiering is disabled,
we should not consider that data as special by marking it as hot, correct?
Instead, it should be considered as
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
vinayakphegde commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2037647435 > Overall, looks good, I just think we need to add javadoc on public methods and explain properly when we will consider a file/block as cold/hot, specially when dealing with default tiering type. Sure, We'll add that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
wchevreuil commented on code in PR #5793:
URL: https://github.com/apache/hbase/pull/5793#discussion_r1551838394
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+return !dataTieringType.equals(DataTieringType.NONE);
+ }
+
+ public boolean isHotData(BlockCacheKey key) throws DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isHotData(hFilePath);
+ }
+
+ public boolean isHotData(Path hFilePath) throws DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType = getDataTieringType(configuration);
+
+if (dataTieringType.equals(DataTieringType.TIME_RANGE)) {
+ long hotDataAge = getDataTieringHotDataAge(configuration);
+
+ HStoreFile hStoreFile = getHStoreFile(hFilePath);
+ if (hStoreFile == null) {
+throw new DataTieringException(
Review Comment:
Rather than throwing exception, maybe just return false? I wonder if it
would be possible that a compaction completed and moved the given file away
just after we entered this method.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *
Re: [PR] HBASE-28465 Implementation of framework for time-based priority bucket-cache [hbase]
Apache-HBase commented on PR #5793: URL: https://github.com/apache/hbase/pull/5793#issuecomment-2037183945 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 40s | HBASE-28463 passed | | +1 :green_heart: | compile | 3m 12s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 42s | HBASE-28463 passed | | +1 :green_heart: | spotless | 0m 58s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 2s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 44s | the patch passed | | +1 :green_heart: | compile | 3m 6s | the patch passed | | +1 :green_heart: | javac | 3m 6s | the patch passed | | -0 :warning: | checkstyle | 0m 46s | hbase-server: The patch generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 6m 24s | Patch does not cause any errors with Hadoop 3.3.6. | | -1 :x: | spotless | 0m 47s | patch has 46 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 2m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 37m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5793 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux d3046cc1445b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / 28c1e3b2a6 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/1/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 77 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5793/1/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
