Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
wchevreuil merged PR #5829: URL: https://github.com/apache/hbase/pull/5829 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2075626000 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 15s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 23s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 56s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 6m 22s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | HBASE-28463 passed | | -0 :warning: | patch | 7m 1s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 21s | the patch passed | | +1 :green_heart: | compile | 0m 54s | the patch passed | | +1 :green_heart: | javac | 0m 54s | the patch passed | | +1 :green_heart: | shadedjars | 6m 15s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 29s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 264m 3s | hbase-server in the patch failed. | | | | 292m 5s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/12/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 459c3541b08b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/12/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/12/testReport/ | | Max. process+thread count | 4564 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/12/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2075107311 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 40s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 29s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 44s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 27s | the patch passed | | +1 :green_heart: | compile | 0m 42s | the patch passed | | +1 :green_heart: | javac | 0m 42s | the patch passed | | +1 :green_heart: | shadedjars | 5m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 11m 32s | hbase-server in the patch failed. | | | | 31m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/12/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 913a46bdbdc7 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/12/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/12/testReport/ | | Max. process+thread count | 1578 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/12/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2075100076 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 1s | HBASE-28463 passed | | +1 :green_heart: | compile | 2m 30s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 39s | HBASE-28463 passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 34s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 44s | the patch passed | | +1 :green_heart: | compile | 2m 31s | the patch passed | | +1 :green_heart: | javac | 2m 31s | the patch passed | | -0 :warning: | checkstyle | 0m 38s | hbase-server: The patch generated 1 new + 49 unchanged - 0 fixed = 50 total (was 49) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 4m 50s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 28m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/12/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux f8e926b700bb 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/12/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 80 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/12/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
wchevreuil commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1577636485
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -245,6 +248,184 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+assert
(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity. The hot file
+ * remains in the cache.
+ */
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new BlockCacheKey(hStoreFiles.get(2).getPath(), 0,
true, BlockType.DATA);
+CacheTestUtils.HFileBlockPair[] newBlock =
CacheTestUtils.generateHFileBlocks(8192, 1);
+
+bucketCache.cacheBlock(newKey, newBlock[0].getBlock());
+Waiter.waitFor(defaultConf, 100, 100,
+ () -> (bucketCache.getBackingMap().containsKey(newKey)));
+
+// Verify that the bucket cache now contains 2 hot blocks blocks only.
+// Both cold blocks of 8KB will be evicted to make room for 1 block of 8KB
+ an additional
+// space.
+validateBlocks(bucketCache.getBackingMap().keySet(), 2, 2, 0);
Review Comment:
Yeah, we had reviewed this internally, not at all an issue. It's due to the
internals of BucketAllocator when calculating buckets limits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2074021885 retest -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1577210450
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -245,6 +248,184 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+assert
(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity. The hot file
+ * remains in the cache.
+ */
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new BlockCacheKey(hStoreFiles.get(2).getPath(), 0,
true, BlockType.DATA);
+CacheTestUtils.HFileBlockPair[] newBlock =
CacheTestUtils.generateHFileBlocks(8192, 1);
+
+bucketCache.cacheBlock(newKey, newBlock[0].getBlock());
+Waiter.waitFor(defaultConf, 100, 100,
+ () -> (bucketCache.getBackingMap().containsKey(newKey)));
+
+// Verify that the bucket cache now contains 2 hot blocks blocks only.
+// Both cold blocks of 8KB will be evicted to make room for 1 block of 8KB
+ an additional
+// space.
+validateBlocks(bucketCache.getBackingMap().keySet(), 2, 2, 0);
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity, but one cold
+ * block remains in the cache since the required space is freed.
+ */
+ @Test
+ public void testBlockEvictionsAllColdBlocks() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with three cold data blocks.
+// hStoreFiles.get(3) is a cold data file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 16384, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1577210450
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -245,6 +248,184 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+assert
(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity. The hot file
+ * remains in the cache.
+ */
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new BlockCacheKey(hStoreFiles.get(2).getPath(), 0,
true, BlockType.DATA);
+CacheTestUtils.HFileBlockPair[] newBlock =
CacheTestUtils.generateHFileBlocks(8192, 1);
+
+bucketCache.cacheBlock(newKey, newBlock[0].getBlock());
+Waiter.waitFor(defaultConf, 100, 100,
+ () -> (bucketCache.getBackingMap().containsKey(newKey)));
+
+// Verify that the bucket cache now contains 2 hot blocks blocks only.
+// Both cold blocks of 8KB will be evicted to make room for 1 block of 8KB
+ an additional
+// space.
+validateBlocks(bucketCache.getBackingMap().keySet(), 2, 2, 0);
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity, but one cold
+ * block remains in the cache since the required space is freed.
+ */
+ @Test
+ public void testBlockEvictionsAllColdBlocks() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with three cold data blocks.
+// hStoreFiles.get(3) is a cold data file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 16384, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1577210450
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -245,6 +248,184 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+assert
(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity. The hot file
+ * remains in the cache.
+ */
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new BlockCacheKey(hStoreFiles.get(2).getPath(), 0,
true, BlockType.DATA);
+CacheTestUtils.HFileBlockPair[] newBlock =
CacheTestUtils.generateHFileBlocks(8192, 1);
+
+bucketCache.cacheBlock(newKey, newBlock[0].getBlock());
+Waiter.waitFor(defaultConf, 100, 100,
+ () -> (bucketCache.getBackingMap().containsKey(newKey)));
+
+// Verify that the bucket cache now contains 2 hot blocks blocks only.
+// Both cold blocks of 8KB will be evicted to make room for 1 block of 8KB
+ an additional
+// space.
+validateBlocks(bucketCache.getBackingMap().keySet(), 2, 2, 0);
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity, but one cold
+ * block remains in the cache since the required space is freed.
+ */
+ @Test
+ public void testBlockEvictionsAllColdBlocks() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with three cold data blocks.
+// hStoreFiles.get(3) is a cold data file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 16384, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1577210450
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -245,6 +248,184 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+assert
(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity. The hot file
+ * remains in the cache.
+ */
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new BlockCacheKey(hStoreFiles.get(2).getPath(), 0,
true, BlockType.DATA);
+CacheTestUtils.HFileBlockPair[] newBlock =
CacheTestUtils.generateHFileBlocks(8192, 1);
+
+bucketCache.cacheBlock(newKey, newBlock[0].getBlock());
+Waiter.waitFor(defaultConf, 100, 100,
+ () -> (bucketCache.getBackingMap().containsKey(newKey)));
+
+// Verify that the bucket cache now contains 2 hot blocks blocks only.
+// Both cold blocks of 8KB will be evicted to make room for 1 block of 8KB
+ an additional
+// space.
+validateBlocks(bucketCache.getBackingMap().keySet(), 2, 2, 0);
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity, but one cold
+ * block remains in the cache since the required space is freed.
+ */
+ @Test
+ public void testBlockEvictionsAllColdBlocks() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with three cold data blocks.
+// hStoreFiles.get(3) is a cold data file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 16384, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2073543053 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 30s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 43s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 22s | the patch passed | | +1 :green_heart: | compile | 0m 44s | the patch passed | | +1 :green_heart: | javac | 0m 44s | the patch passed | | +1 :green_heart: | shadedjars | 5m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 287m 8s | hbase-server in the patch failed. | | | | 310m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/11/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 72fafacd2753 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/11/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/11/testReport/ | | Max. process+thread count | 5219 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/11/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2073438892 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 41s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 46s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 37s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 43s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed | | +1 :green_heart: | javac | 0m 45s | the patch passed | | +1 :green_heart: | shadedjars | 5m 35s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 229m 28s | hbase-server in the patch failed. | | | | 252m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/11/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f2ab4fa85a6c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/11/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/11/testReport/ | | Max. process+thread count | 4881 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/11/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2072995664 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 15s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 23s | HBASE-28463 passed | | +1 :green_heart: | compile | 3m 56s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 49s | HBASE-28463 passed | | +1 :green_heart: | spotless | 0m 51s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 24s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 3m 33s | the patch passed | | +1 :green_heart: | javac | 3m 33s | the patch passed | | -0 :warning: | checkstyle | 0m 46s | hbase-server: The patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 7m 11s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 1m 11s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 41m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/11/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux f614a268dafd 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/11/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/11/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2072910052 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 25s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 41s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 27s | the patch passed | | +1 :green_heart: | compile | 0m 42s | the patch passed | | +1 :green_heart: | javac | 0m 42s | the patch passed | | +1 :green_heart: | shadedjars | 5m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 244m 29s | hbase-server in the patch failed. | | | | 267m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/10/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2190d396b05e 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/10/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/10/testReport/ | | Max. process+thread count | 4660 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/10/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2072880752 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 12s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 45s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 40s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 47s | the patch passed | | +1 :green_heart: | compile | 0m 46s | the patch passed | | +1 :green_heart: | javac | 0m 46s | the patch passed | | +1 :green_heart: | shadedjars | 5m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 227m 16s | hbase-server in the patch passed. | | | | 251m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/10/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f07b34d2f353 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/10/testReport/ | | Max. process+thread count | 5152 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/10/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1576267214
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -245,6 +248,184 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+assert
(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity. The hot file
+ * remains in the cache.
+ */
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new BlockCacheKey(hStoreFiles.get(2).getPath(), 0,
true, BlockType.DATA);
+CacheTestUtils.HFileBlockPair[] newBlock =
CacheTestUtils.generateHFileBlocks(8192, 1);
+
+bucketCache.cacheBlock(newKey, newBlock[0].getBlock());
+Waiter.waitFor(defaultConf, 100, 100,
+ () -> (bucketCache.getBackingMap().containsKey(newKey)));
+
+// Verify that the bucket cache now contains 2 hot blocks blocks only.
+// Both cold blocks of 8KB will be evicted to make room for 1 block of 8KB
+ an additional
+// space.
+validateBlocks(bucketCache.getBackingMap().keySet(), 2, 2, 0);
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity, but one cold
+ * block remains in the cache since the required space is freed.
+ */
+ @Test
+ public void testBlockEvictionsAllColdBlocks() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with three cold data blocks.
+// hStoreFiles.get(3) is a cold data file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 16384, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2072212903 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 12s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | HBASE-28463 passed | | +1 :green_heart: | compile | 2m 32s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 37s | HBASE-28463 passed | | +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 32s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | compile | 2m 30s | the patch passed | | +1 :green_heart: | javac | 2m 30s | the patch passed | | -0 :warning: | checkstyle | 0m 37s | hbase-server: The patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 4m 45s | Patch does not cause any errors with Hadoop 3.3.6. | | -1 :x: | spotless | 0m 38s | patch has 43 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 27m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/10/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux fc54dbb65055 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/10/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/10/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 80 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/10/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1576153194
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -245,6 +248,184 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+assert
(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity. The hot file
+ * remains in the cache.
+ */
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new BlockCacheKey(hStoreFiles.get(2).getPath(), 0,
true, BlockType.DATA);
+CacheTestUtils.HFileBlockPair[] newBlock =
CacheTestUtils.generateHFileBlocks(8192, 1);
+
+bucketCache.cacheBlock(newKey, newBlock[0].getBlock());
+Waiter.waitFor(defaultConf, 100, 100,
+ () -> (bucketCache.getBackingMap().containsKey(newKey)));
+
+// Verify that the bucket cache now contains 2 hot blocks blocks only.
+// Both cold blocks of 8KB will be evicted to make room for 1 block of 8KB
+ an additional
+// space.
+validateBlocks(bucketCache.getBackingMap().keySet(), 2, 2, 0);
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity, but one cold
+ * block remains in the cache since the required space is freed.
+ */
+ @Test
+ public void testBlockEvictionsAllColdBlocks() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with three cold data blocks.
+// hStoreFiles.get(3) is a cold data file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 16384, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1576108275
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -245,6 +248,184 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+assert
(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity. The hot file
+ * remains in the cache.
+ */
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new BlockCacheKey(hStoreFiles.get(2).getPath(), 0,
true, BlockType.DATA);
+CacheTestUtils.HFileBlockPair[] newBlock =
CacheTestUtils.generateHFileBlocks(8192, 1);
+
+bucketCache.cacheBlock(newKey, newBlock[0].getBlock());
+Waiter.waitFor(defaultConf, 100, 100,
+ () -> (bucketCache.getBackingMap().containsKey(newKey)));
+
+// Verify that the bucket cache now contains 2 hot blocks blocks only.
+// Both cold blocks of 8KB will be evicted to make room for 1 block of 8KB
+ an additional
+// space.
+validateBlocks(bucketCache.getBackingMap().keySet(), 2, 2, 0);
Review Comment:
During my debugging I found that eviction does get triggered. However, I did
reduce the size to 40K and the eviction still triggers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1576096528
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -245,6 +248,184 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+assert
(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity. The hot file
+ * remains in the cache.
+ */
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
Review Comment:
Yes, you are right. Let me bring it down to 10 secs timeout. I had set it to
a high value while debugging it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
wchevreuil commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1576059968
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -245,6 +248,184 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+assert
(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity. The hot file
+ * remains in the cache.
+ */
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
Review Comment:
Should we fail faster? Why this timeout needs to be so long?
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -245,6 +248,184 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+assert
(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ /*
+ * Verify that two cold blocks(both) are evicted when bucket reaches its
capacity. The hot file
+ * remains in the cache.
+ */
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+Set cacheKeys = new HashSet<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for (BlockCacheKey key : cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ Waiter.waitFor(defaultConf, 100, 100,
+() -> (bucketCache.getBackingMap().containsKey(key)));
+}
+
+// Verify that the bucket cache contains 3 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new BlockCacheKey(hStoreFiles.get(2).getPath(), 0,
true, BlockType.DATA);
+CacheTestUtils.HFileBlockPair[] newBlock =
CacheTestUtils.generateHFileBlocks(8192, 1);
+
+bucketCache.cacheBlock(newKey, newBlock[0].getBlock());
+Waiter.waitFor(defaultConf, 100, 100,
+ () ->
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2071092582 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 44s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 40s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 27s | the patch passed | | +1 :green_heart: | compile | 0m 39s | the patch passed | | +1 :green_heart: | javac | 0m 39s | the patch passed | | +1 :green_heart: | shadedjars | 5m 43s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 239m 7s | hbase-server in the patch passed. | | | | 262m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux f3e44c8b0fb9 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/9/testReport/ | | Max. process+thread count | 5372 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/9/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2070706655 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 17s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | HBASE-28463 passed | | +1 :green_heart: | compile | 2m 40s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 45s | HBASE-28463 passed | | +1 :green_heart: | spotless | 0m 47s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 36s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 43s | the patch passed | | +1 :green_heart: | compile | 3m 32s | the patch passed | | +1 :green_heart: | javac | 3m 32s | the patch passed | | -0 :warning: | checkstyle | 0m 43s | hbase-server: The patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 6m 12s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 43s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 10s | The patch does not generate ASF License warnings. | | | | 33m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/9/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 6d01abe693f9 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/9/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 82 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/9/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2070581974 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 52s | HBASE-28463 passed | | +1 :green_heart: | compile | 1m 6s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 6m 54s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 41s | the patch passed | | +1 :green_heart: | compile | 1m 0s | the patch passed | | +1 :green_heart: | javac | 1m 0s | the patch passed | | +1 :green_heart: | shadedjars | 6m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 270m 9s | hbase-server in the patch failed. | | | | 299m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/8/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c0c985c7df5d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/8/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/8/testReport/ | | Max. process+thread count | 4635 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/8/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2070475898 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 36s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 42s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 5m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 28s | the patch passed | | +1 :green_heart: | compile | 0m 42s | the patch passed | | +1 :green_heart: | javac | 0m 42s | the patch passed | | +1 :green_heart: | shadedjars | 5m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 246m 56s | hbase-server in the patch failed. | | | | 269m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/8/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 98ce70616485 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/8/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/8/testReport/ | | Max. process+thread count | 4411 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/8/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2069564603 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 12s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 51s | HBASE-28463 passed | | +1 :green_heart: | compile | 2m 31s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 36s | HBASE-28463 passed | | +1 :green_heart: | spotless | 0m 42s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 32s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | compile | 2m 29s | the patch passed | | +1 :green_heart: | javac | 2m 29s | the patch passed | | -0 :warning: | checkstyle | 0m 37s | hbase-server: The patch generated 19 new + 14 unchanged - 0 fixed = 33 total (was 14) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 4m 57s | Patch does not cause any errors with Hadoop 3.3.6. | | -1 :x: | spotless | 0m 36s | patch has 65 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 27m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/8/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 9de76c7a970c 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / a2321ce9d8 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/8/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/8/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 81 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/8/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2069465706 > Please rebase your local branch with the current state of remote [HBASE-28463](https://issues.apache.org/jira/browse/HBASE-28463) then force push your changes to resolve the conflicts. Done! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1574757909
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The DataTieringManager class categorizes data into hot data and cold data
based on the specified
+ * {@link DataTieringType} when DataTiering is enabled. DataTiering is
disabled by default with
+ * {@link DataTieringType} set to {@link DataTieringType#NONE}. The {@link
DataTieringType}
+ * determines the logic for distinguishing data into hot or cold. By default,
all data is considered
+ * as hot.
+ */
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ /**
+ * Initializes the DataTieringManager instance with the provided map of
online regions.
+ * @param onlineRegions A map containing online regions.
+ */
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ /**
+ * Retrieves the instance of DataTieringManager.
+ * @return The instance of DataTieringManager.
+ * @throws IllegalStateException if DataTieringManager has not been
instantiated.
+ */
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ /**
+ * Determines whether data tiering is enabled for the given block cache key.
+ * @param key the block cache key
+ * @return {@code true} if data tiering is enabled for the HFile associated
with the key,
+ * {@code false} otherwise
+ * @throws DataTieringException if there is an error retrieving the HFile
path or configuration
+ */
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ /**
+ * Determines whether data tiering is enabled for the given HFile path.
+ * @param hFilePath the path to the HFile
+ * @return {@code true} if data tiering is enabled, {@code false} otherwise
+ * @throws DataTieringException if there is an error retrieving the
configuration
+ */
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType =
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1574577429
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The DataTieringManager class categorizes data into hot data and cold data
based on the specified
+ * {@link DataTieringType} when DataTiering is enabled. DataTiering is
disabled by default with
+ * {@link DataTieringType} set to {@link DataTieringType#NONE}. The {@link
DataTieringType}
+ * determines the logic for distinguishing data into hot or cold. By default,
all data is considered
+ * as hot.
+ */
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ /**
+ * Initializes the DataTieringManager instance with the provided map of
online regions.
+ * @param onlineRegions A map containing online regions.
+ */
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ /**
+ * Retrieves the instance of DataTieringManager.
+ * @return The instance of DataTieringManager.
+ * @throws IllegalStateException if DataTieringManager has not been
instantiated.
+ */
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ /**
+ * Determines whether data tiering is enabled for the given block cache key.
+ * @param key the block cache key
+ * @return {@code true} if data tiering is enabled for the HFile associated
with the key,
+ * {@code false} otherwise
+ * @throws DataTieringException if there is an error retrieving the HFile
path or configuration
+ */
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ /**
+ * Determines whether data tiering is enabled for the given HFile path.
+ * @param hFilePath the path to the HFile
+ * @return {@code true} if data tiering is enabled, {@code false} otherwise
+ * @throws DataTieringException if there is an error retrieving the
configuration
+ */
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType =
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
wchevreuil commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2069053516 Please rebase your local branch with the current state of remote HBASE-28463 then force push your changes to resolve the conflicts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
wchevreuil commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1574522829
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheKey;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * The DataTieringManager class categorizes data into hot data and cold data
based on the specified
+ * {@link DataTieringType} when DataTiering is enabled. DataTiering is
disabled by default with
+ * {@link DataTieringType} set to {@link DataTieringType#NONE}. The {@link
DataTieringType}
+ * determines the logic for distinguishing data into hot or cold. By default,
all data is considered
+ * as hot.
+ */
+@InterfaceAudience.Private
+public class DataTieringManager {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataTieringManager.class);
+ public static final String DATATIERING_KEY = "hbase.hstore.datatiering.type";
+ public static final String DATATIERING_HOT_DATA_AGE_KEY =
+"hbase.hstore.datatiering.hot.age.millis";
+ public static final DataTieringType DEFAULT_DATATIERING =
DataTieringType.NONE;
+ public static final long DEFAULT_DATATIERING_HOT_DATA_AGE = 7 * 24 * 60 * 60
* 1000; // 7 Days
+ private static DataTieringManager instance;
+ private final Map onlineRegions;
+
+ private DataTieringManager(Map onlineRegions) {
+this.onlineRegions = onlineRegions;
+ }
+
+ /**
+ * Initializes the DataTieringManager instance with the provided map of
online regions.
+ * @param onlineRegions A map containing online regions.
+ */
+ public static synchronized void instantiate(Map
onlineRegions) {
+if (instance == null) {
+ instance = new DataTieringManager(onlineRegions);
+ LOG.info("DataTieringManager instantiated successfully.");
+} else {
+ LOG.warn("DataTieringManager is already instantiated.");
+}
+ }
+
+ /**
+ * Retrieves the instance of DataTieringManager.
+ * @return The instance of DataTieringManager.
+ * @throws IllegalStateException if DataTieringManager has not been
instantiated.
+ */
+ public static synchronized DataTieringManager getInstance() {
+if (instance == null) {
+ throw new IllegalStateException(
+"DataTieringManager has not been instantiated. Call instantiate()
first.");
+}
+return instance;
+ }
+
+ /**
+ * Determines whether data tiering is enabled for the given block cache key.
+ * @param key the block cache key
+ * @return {@code true} if data tiering is enabled for the HFile associated
with the key,
+ * {@code false} otherwise
+ * @throws DataTieringException if there is an error retrieving the HFile
path or configuration
+ */
+ public boolean isDataTieringEnabled(BlockCacheKey key) throws
DataTieringException {
+Path hFilePath = key.getFilePath();
+if (hFilePath == null) {
+ throw new DataTieringException("BlockCacheKey Doesn't Contain HFile
Path");
+}
+return isDataTieringEnabled(hFilePath);
+ }
+
+ /**
+ * Determines whether data tiering is enabled for the given HFile path.
+ * @param hFilePath the path to the HFile
+ * @return {@code true} if data tiering is enabled, {@code false} otherwise
+ * @throws DataTieringException if there is an error retrieving the
configuration
+ */
+ public boolean isDataTieringEnabled(Path hFilePath) throws
DataTieringException {
+Configuration configuration = getConfiguration(hFilePath);
+DataTieringType dataTieringType =
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2067299438 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 45s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 7s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 54s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 6m 31s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 28s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 16s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 6m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 281m 22s | hbase-server in the patch failed. | | | | 308m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 65900ace50fe 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / b7bb8b9d27 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/6/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/6/testReport/ | | Max. process+thread count | 5686 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/6/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2067289117 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 54s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 6m 24s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 34s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 6m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 269m 9s | hbase-server in the patch passed. | | | | 297m 27s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 62c250f92b34 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / b7bb8b9d27 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/6/testReport/ | | Max. process+thread count | 4485 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/6/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2066926092 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 16s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 50s | HBASE-28463 passed | | +1 :green_heart: | compile | 2m 29s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 37s | HBASE-28463 passed | | +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 39s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | compile | 2m 28s | the patch passed | | +1 :green_heart: | javac | 2m 28s | the patch passed | | -0 :warning: | checkstyle | 0m 36s | hbase-server: The patch generated 22 new + 14 unchanged - 0 fixed = 36 total (was 14) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 4m 49s | Patch does not cause any errors with Hadoop 3.3.6. | | -1 :x: | spotless | 0m 37s | patch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 1m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 11s | The patch does not generate ASF License warnings. | | | | 27m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 8b3a63ff20c5 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / b7bb8b9d27 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/6/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 80 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/6/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2066884942 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 21s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 58s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 6m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 54s | the patch passed | | +1 :green_heart: | compile | 0m 50s | the patch passed | | +1 :green_heart: | javac | 0m 50s | the patch passed | | +1 :green_heart: | shadedjars | 6m 2s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 294m 32s | hbase-server in the patch failed. | | | | 321m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux da369e1e1e78 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / b7bb8b9d27 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/5/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/5/testReport/ | | Max. process+thread count | 4608 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2066842002 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 12s | HBASE-28463 passed | | +1 :green_heart: | compile | 1m 1s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 6m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 30s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 16s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | shadedjars | 6m 24s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 267m 12s | hbase-server in the patch failed. | | | | 295m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6149f488d587 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / b7bb8b9d27 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/5/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/5/testReport/ | | Max. process+thread count | 4743 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/5/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1572313499
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -999,6 +1019,12 @@ void freeSpace(final String why) {
}
}
+ // Check if the cold file eviction is sufficient to create enough space.
Review Comment:
Ok, given that we do not want to delay the current caching thread any
further once the required space is available. This will also avoid misses on
the remaining cold blocks. Hence, I think, return as soon as the the required
space is available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1572311160
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -218,6 +225,65 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+
assert(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+List cacheKeys = new ArrayList<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for(BlockCacheKey key: cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ while (!(bucketCache.getBackingMap().containsKey(key))) {
+Thread.sleep(100);
Review Comment:
ack!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1572310939
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -218,6 +225,65 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+
assert(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+List cacheKeys = new ArrayList<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for(BlockCacheKey key: cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ while (!(bucketCache.getBackingMap().containsKey(key))) {
+Thread.sleep(100);
+ }
+}
+
+// Verify that the bucket cache contains 4 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
+
+// Add an additional block into cache with hot data which should trigger
the eviction
+BlockCacheKey newKey = new BlockCacheKey(hStoreFiles.get(2).getPath(), 0,
true, BlockType.DATA);
+CacheTestUtils.HFileBlockPair[] newBlock =
CacheTestUtils.generateHFileBlocks(8192, 1);
+
+bucketCache.cacheBlock(newKey, newBlock[0].getBlock());
+while (!(bucketCache.getBackingMap().containsKey(newKey))) {
+ Thread.sleep(100);
+}
Review Comment:
ack!
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -218,6 +225,65 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+
assert(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+List cacheKeys = new ArrayList<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for(BlockCacheKey key: cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ while (!(bucketCache.getBackingMap().containsKey(key))) {
+Thread.sleep(100);
+ }
+}
+
+// Verify that the bucket cache contains 4 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
Review Comment:
ack!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
wchevreuil commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1572259557
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -999,6 +1019,12 @@ void freeSpace(final String why) {
}
}
+ // Check if the cold file eviction is sufficient to create enough space.
Review Comment:
I don't know, just wanted to make sure we considered the two options. If we
delete all cold files even if that meant we will free more than the originally
wanted free bytes, we might delay further mass evictions, at the cost of having
more misses if these cold blocks are still accessed. I leave the final decision
to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
wchevreuil commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1572252207
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -218,6 +225,65 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+
assert(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+List cacheKeys = new ArrayList<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for(BlockCacheKey key: cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ while (!(bucketCache.getBackingMap().containsKey(key))) {
+Thread.sleep(100);
Review Comment:
nit: use `Waiter.waitFor`
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -218,6 +225,65 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+
assert(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with lower capacity
+BucketCache bucketCache = new BucketCache("file:" + testDir +
"/bucket.cache", capacitySize,
+ 8192, bucketSizes, writeThreads, writerQLen, testDir +
"/bucket.persistence",
+ DEFAULT_ERROR_TOLERATION_DURATION, defaultConf);
+
+// Create three Cache keys with cold data files and a block with hot data.
+// hStoreFiles.get(3) is a cold data file, while hStoreFiles.get(0) is a
hot file.
+List cacheKeys = new ArrayList<>();
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 0, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(3).getPath(), 8192, true,
BlockType.DATA));
+cacheKeys.add(new BlockCacheKey(hStoreFiles.get(0).getPath(), 0, true,
BlockType.DATA));
+
+// Create dummy data to be cached and fill the cache completely.
+CacheTestUtils.HFileBlockPair[] blocks =
CacheTestUtils.generateHFileBlocks(8192, 3);
+
+int blocksIter = 0;
+for(BlockCacheKey key: cacheKeys) {
+ bucketCache.cacheBlock(key, blocks[blocksIter++].getBlock());
+ // Ensure that the block is persisted to the file.
+ while (!(bucketCache.getBackingMap().containsKey(key))) {
+Thread.sleep(100);
+ }
+}
+
+// Verify that the bucket cache contains 4 blocks.
+assertEquals(3, bucketCache.getBackingMap().keySet().size());
Review Comment:
nit: comment mismatching the code.
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDataTieringManager.java:
##
@@ -218,6 +225,65 @@ public void testColdDataFiles() {
}
}
+ @Test
+ public void testPickColdDataFiles() {
+Map coldDataFiles = dataTieringManager.getColdFilesList();
+assertEquals(1, coldDataFiles.size());
+// hStoreFiles[3] is the cold file.
+
assert(coldDataFiles.containsKey(hStoreFiles.get(3).getFileInfo().getActiveFileName()));
+ }
+
+ @Test
+ public void testBlockEvictions() throws Exception {
+long capacitySize = 64 * 1024;
+int writeThreads = 3;
+int writerQLen = 64;
+int[] bucketSizes = new int[] { 8 * 1024 + 1024 };
+
+// Setup: Create a bucket cache with
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1572253832
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -983,6 +994,15 @@ void freeSpace(final String why) {
// Scan entire map putting bucket entry into appropriate bucket entry
// group
for (Map.Entry bucketEntryWithKey :
backingMap.entrySet()) {
+
+if (coldFiles != null
+ &&
coldFiles.containsKey(bucketEntryWithKey.getKey().getHfileName())) {
+ int freedBlockSize = bucketEntryWithKey.getValue().getLength();
+ evictBlockIfNoRpcReferenced(bucketEntryWithKey.getKey());
Review Comment:
ack!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1572253085
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -999,6 +1019,12 @@ void freeSpace(final String why) {
}
}
+ // Check if the cold file eviction is sufficient to create enough space.
Review Comment:
@wchevreuil, right now in the code, I am evicting all the cold files and
then checking if we have more bytes to free.
Would it make sense to do it like this or do we want to evict only upto
bytesToFreeWithExtra and exit leaving behind some of the cold blocks?
Which one would be preferable? The later option seems user friendly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
wchevreuil commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1572237627
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -983,6 +994,15 @@ void freeSpace(final String why) {
// Scan entire map putting bucket entry into appropriate bucket entry
// group
for (Map.Entry bucketEntryWithKey :
backingMap.entrySet()) {
+
+if (coldFiles != null
+ &&
coldFiles.containsKey(bucketEntryWithKey.getKey().getHfileName())) {
+ int freedBlockSize = bucketEntryWithKey.getValue().getLength();
+ evictBlockIfNoRpcReferenced(bucketEntryWithKey.getKey());
Review Comment:
`evictBlockIfNoRpcReferenced` returns a boolean indicating whether the
eviction was successful or not. We should check that boolean in orther to
decide if we should increment `bytesFreed`.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -980,9 +991,18 @@ void freeSpace(final String why) {
BucketEntryGroup bucketMemory =
new BucketEntryGroup(bytesToFreeWithExtra, blockSize,
getPartitionSize(memoryFactor));
+ long bytesFreed = 0;
+
// Scan entire map putting bucket entry into appropriate bucket entry
// group
for (Map.Entry bucketEntryWithKey :
backingMap.entrySet()) {
+
+if (coldFiles.containsKey(bucketEntryWithKey.getKey().getHfileName()))
{
+ //coldBlocks.add(bucketEntryWithKey.getKey());
+ bytesFreed += bucketEntryWithKey.getValue().getLength();
+ evictBlock(bucketEntryWithKey.getKey());
Review Comment:
No, that not what I meant. You already have `bytesToFreeWithExtra`
calculated in line #983 and you are computing `bytesFreed`. Once `bytesFreed`
reached `bytesToFreeWithExtra` value, you still want to evict the cold files or
should we just interrupt the loop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2066379398 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 15s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 18s | HBASE-28463 passed | | +1 :green_heart: | compile | 3m 13s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 37s | HBASE-28463 passed | | +1 :green_heart: | spotless | 0m 46s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 51s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 50s | the patch passed | | +1 :green_heart: | compile | 3m 18s | the patch passed | | +1 :green_heart: | javac | 3m 18s | the patch passed | | -0 :warning: | checkstyle | 0m 45s | hbase-server: The patch generated 20 new + 14 unchanged - 0 fixed = 34 total (was 14) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 36s | Patch does not cause any errors with Hadoop 3.3.6. | | -1 :x: | spotless | 1m 4s | patch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 3m 3s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 20s | The patch does not generate ASF License warnings. | | | | 38m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux afc6055f5c71 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / b7bb8b9d27 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/5/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 78 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/5/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1572161825
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -980,9 +991,18 @@ void freeSpace(final String why) {
BucketEntryGroup bucketMemory =
new BucketEntryGroup(bytesToFreeWithExtra, blockSize,
getPartitionSize(memoryFactor));
+ long bytesFreed = 0;
+
// Scan entire map putting bucket entry into appropriate bucket entry
// group
for (Map.Entry bucketEntryWithKey :
backingMap.entrySet()) {
+
+if (coldFiles.containsKey(bucketEntryWithKey.getKey().getHfileName()))
{
+ //coldBlocks.add(bucketEntryWithKey.getKey());
+ bytesFreed += bucketEntryWithKey.getValue().getLength();
+ evictBlock(bucketEntryWithKey.getKey());
Review Comment:
You mean we need to evict these cold file blocks even before the
calculations for bytesToFreeWithoutExtra? That will require one additional
traversal of backingMap. I will incorporate this change and we can take a look
if that looks ok.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -219,4 +225,43 @@ private long getDataTieringHotDataAge(Configuration conf) {
return Long.parseLong(
conf.get(DATATIERING_HOT_DATA_AGE_KEY,
String.valueOf(DEFAULT_DATATIERING_HOT_DATA_AGE)));
}
+
+ /*
+ * This API takes the names of files as input and returns a subset of these
file names
+ * that are cold.
+ * @parameter inputFileNames: Input list of file names
+ * @return List of names of files that are cold as per data-tiering logic.
+ */
+ public Map getColdFilesList() {
+Map coldFiles = new HashMap<>();
+for (HRegion r : this.onlineRegions.values()) {
+ for (HStore hStore : r.getStores()) {
+Configuration conf = hStore.getReadOnlyConfiguration();
+if (getDataTieringType(conf) != DataTieringType.TIME_RANGE) {
+ // Data-Tiering not enabled for the store. Just skip it.
+ continue;
+}
+Long hotDataAge = getDataTieringHotDataAge(conf);
+
+for (HStoreFile hStoreFile : hStore.getStorefiles()) {
+ String hFileName =
+
hStoreFile.getFileInfo().getHFileInfo().getHFileContext().getHFileName();
+OptionalLong maxTimestamp = hStoreFile.getMaximumTimestamp();
+if (!maxTimestamp.isPresent()) {
+ // We could throw from here, But we are already in the critical
code-path
+ // of freeing space. Hence, we can ignore that file for now
+ // Or do we want to include it?
Review Comment:
ack!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1572161068
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -980,9 +991,18 @@ void freeSpace(final String why) {
BucketEntryGroup bucketMemory =
new BucketEntryGroup(bytesToFreeWithExtra, blockSize,
getPartitionSize(memoryFactor));
+ long bytesFreed = 0;
+
// Scan entire map putting bucket entry into appropriate bucket entry
// group
for (Map.Entry bucketEntryWithKey :
backingMap.entrySet()) {
+
+if (coldFiles.containsKey(bucketEntryWithKey.getKey().getHfileName()))
{
+ //coldBlocks.add(bucketEntryWithKey.getKey());
Review Comment:
ack!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
jhungund commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1572155034
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -980,9 +991,18 @@ void freeSpace(final String why) {
BucketEntryGroup bucketMemory =
new BucketEntryGroup(bytesToFreeWithExtra, blockSize,
getPartitionSize(memoryFactor));
+ long bytesFreed = 0;
+
// Scan entire map putting bucket entry into appropriate bucket entry
// group
for (Map.Entry bucketEntryWithKey :
backingMap.entrySet()) {
+
+if (coldFiles.containsKey(bucketEntryWithKey.getKey().getHfileName()))
{
+ //coldBlocks.add(bucketEntryWithKey.getKey());
+ bytesFreed += bucketEntryWithKey.getValue().getLength();
Review Comment:
ack!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2064931623 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 15s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 22s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 52s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 6m 18s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 31s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 20s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 5m 41s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 27s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 341m 47s | hbase-server in the patch failed. | | | | 369m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c3c83ad34e25 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / b7bb8b9d27 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/4/testReport/ | | Max. process+thread count | 4771 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2064832622 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 45s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 23s | HBASE-28463 passed | | +1 :green_heart: | compile | 0m 51s | HBASE-28463 passed | | +1 :green_heart: | shadedjars | 6m 34s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 33s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 16s | the patch passed | | +1 :green_heart: | compile | 0m 53s | the patch passed | | +1 :green_heart: | javac | 0m 53s | the patch passed | | +1 :green_heart: | shadedjars | 5m 55s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 310m 37s | hbase-server in the patch failed. | | | | 339m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 83a9678c4ce5 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / b7bb8b9d27 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/4/testReport/ | | Max. process+thread count | 4948 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/4/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
wchevreuil commented on code in PR #5829:
URL: https://github.com/apache/hbase/pull/5829#discussion_r1570904381
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DataTieringManager.java:
##
@@ -219,4 +225,43 @@ private long getDataTieringHotDataAge(Configuration conf) {
return Long.parseLong(
conf.get(DATATIERING_HOT_DATA_AGE_KEY,
String.valueOf(DEFAULT_DATATIERING_HOT_DATA_AGE)));
}
+
+ /*
+ * This API takes the names of files as input and returns a subset of these
file names
+ * that are cold.
+ * @parameter inputFileNames: Input list of file names
+ * @return List of names of files that are cold as per data-tiering logic.
+ */
+ public Map getColdFilesList() {
+Map coldFiles = new HashMap<>();
+for (HRegion r : this.onlineRegions.values()) {
+ for (HStore hStore : r.getStores()) {
+Configuration conf = hStore.getReadOnlyConfiguration();
+if (getDataTieringType(conf) != DataTieringType.TIME_RANGE) {
+ // Data-Tiering not enabled for the store. Just skip it.
+ continue;
+}
+Long hotDataAge = getDataTieringHotDataAge(conf);
+
+for (HStoreFile hStoreFile : hStore.getStorefiles()) {
+ String hFileName =
+
hStoreFile.getFileInfo().getHFileInfo().getHFileContext().getHFileName();
+OptionalLong maxTimestamp = hStoreFile.getMaximumTimestamp();
+if (!maxTimestamp.isPresent()) {
+ // We could throw from here, But we are already in the critical
code-path
+ // of freeing space. Hence, we can ignore that file for now
+ // Or do we want to include it?
Review Comment:
That's fine for me. Can we add a debug/trace log?
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -972,6 +976,13 @@ void freeSpace(final String why) {
long bytesToFreeWithExtra =
(long) Math.floor(bytesToFreeWithoutExtra * (1 + extraFreeFactor));
+ // Check the list of files to determine the cold files which can be
readily evicted.
+ Map coldFiles =
+DataTieringManager.getInstance().getColdFilesList();
+ //Can we evict the blocks inline during backingMap traversal?
+ // If not, we accumulate the keys and evict them later.
+ // List coldBlocks = new ArrayList<>();
+
Review Comment:
nit: please remove commented code.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -980,9 +991,18 @@ void freeSpace(final String why) {
BucketEntryGroup bucketMemory =
new BucketEntryGroup(bytesToFreeWithExtra, blockSize,
getPartitionSize(memoryFactor));
+ long bytesFreed = 0;
+
// Scan entire map putting bucket entry into appropriate bucket entry
// group
for (Map.Entry bucketEntryWithKey :
backingMap.entrySet()) {
+
+if (coldFiles.containsKey(bucketEntryWithKey.getKey().getHfileName()))
{
+ //coldBlocks.add(bucketEntryWithKey.getKey());
Review Comment:
nit: remove commented code.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -980,9 +991,18 @@ void freeSpace(final String why) {
BucketEntryGroup bucketMemory =
new BucketEntryGroup(bytesToFreeWithExtra, blockSize,
getPartitionSize(memoryFactor));
+ long bytesFreed = 0;
+
// Scan entire map putting bucket entry into appropriate bucket entry
// group
for (Map.Entry bucketEntryWithKey :
backingMap.entrySet()) {
+
+if (coldFiles.containsKey(bucketEntryWithKey.getKey().getHfileName()))
{
+ //coldBlocks.add(bucketEntryWithKey.getKey());
+ bytesFreed += bucketEntryWithKey.getValue().getLength();
Review Comment:
We should only account for this once the eviction was indeed successfull.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -935,6 +937,7 @@ void freeSpace(final String why) {
}
try {
freeInProgress = true;
+
Review Comment:
nit: please avoid unnecessary new lines.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java:
##
@@ -980,9 +991,18 @@ void freeSpace(final String why) {
BucketEntryGroup bucketMemory =
new BucketEntryGroup(bytesToFreeWithExtra, blockSize,
getPartitionSize(memoryFactor));
+ long bytesFreed = 0;
+
// Scan entire map putting bucket entry into appropriate bucket entry
// group
for (Map.Entry bucketEntryWithKey :
backingMap.entrySet()) {
+
+if (coldFiles.containsKey(bucketEntryWithKey.getKey().getHfileName()))
{
+ //coldBlocks.add(bucketEntryWithKey.getKey());
+ bytesFreed += bucketEntryWithKey.getValue().getLength();
+ evictBlock(bucketEntryWithKey.getKey());
Review Comment:
Re: [PR] HBASE-28468: Integrate the data-tiering logic into cache evictions. [hbase]
Apache-HBase commented on PR #5829: URL: https://github.com/apache/hbase/pull/5829#issuecomment-2063859032 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ HBASE-28463 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 43s | HBASE-28463 passed | | +1 :green_heart: | compile | 4m 48s | HBASE-28463 passed | | +1 :green_heart: | checkstyle | 0m 59s | HBASE-28463 passed | | +1 :green_heart: | spotless | 1m 23s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 46s | HBASE-28463 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 56s | the patch passed | | +1 :green_heart: | compile | 3m 35s | the patch passed | | +1 :green_heart: | javac | 3m 35s | the patch passed | | -0 :warning: | checkstyle | 0m 57s | hbase-server: The patch generated 19 new + 14 unchanged - 0 fixed = 33 total (was 14) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 58s | Patch does not cause any errors with Hadoop 3.3.6. | | -1 :x: | spotless | 0m 53s | patch has 66 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 2m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 43m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5829 | | JIRA Issue | HBASE-28468 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 6ad7ab8b4f22 5.4.0-176-generic #196-Ubuntu SMP Fri Mar 22 16:46:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-28463 / b7bb8b9d27 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/4/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 82 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5829/4/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
