Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2116939444 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 46s | master passed | | +1 :green_heart: | compile | 1m 28s | master passed | | +1 :green_heart: | shadedjars | 6m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | master passed | | -0 :warning: | patch | 7m 38s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 25s | the patch passed | | +1 :green_heart: | compile | 1m 25s | the patch passed | | +1 :green_heart: | javac | 1m 25s | the patch passed | | +1 :green_heart: | shadedjars | 6m 28s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 268m 25s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 17m 5s | hbase-mapreduce in the patch passed. | | | | 316m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/34/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 44673d172319 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 716adf50e9 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/34/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/34/testReport/ | | Max. process+thread count | 4806 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/34/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2116873570 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 54s | master passed | | +1 :green_heart: | compile | 1m 18s | master passed | | +1 :green_heart: | shadedjars | 5m 19s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | | -0 :warning: | patch | 6m 21s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 51s | the patch passed | | +1 :green_heart: | compile | 1m 17s | the patch passed | | +1 :green_heart: | javac | 1m 17s | the patch passed | | +1 :green_heart: | shadedjars | 5m 19s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 226m 57s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 16m 15s | hbase-mapreduce in the patch passed. | | | | 269m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/34/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e01eef41cba4 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 716adf50e9 | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/34/testReport/ | | Max. process+thread count | 5420 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/34/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
apurtell merged PR #5545: URL: https://github.com/apache/hbase/pull/5545 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2116540777 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 56s | master passed | | +1 :green_heart: | compile | 3m 13s | master passed | | +1 :green_heart: | checkstyle | 0m 52s | master passed | | +1 :green_heart: | spotless | 0m 44s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 2s | master passed | | -0 :warning: | patch | 1m 42s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 56s | the patch passed | | +1 :green_heart: | compile | 4m 20s | the patch passed | | +1 :green_heart: | javac | 4m 20s | the patch passed | | -0 :warning: | checkstyle | 0m 47s | hbase-server: The patch generated 1 new + 46 unchanged - 0 fixed = 47 total (was 46) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 47s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 58s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 54s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 18s | The patch does not generate ASF License warnings. | | | | 37m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/34/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux ec4f2513565d 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 716adf50e9 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/34/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/34/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1604284847
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileWriter.java:
##
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder.NEW_VERSION_BEHAVIOR;
+import static
org.apache.hadoop.hbase.regionserver.StoreFileWriter.ENABLE_HISTORICAL_COMPACTION_FILES;
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Random;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.KeepDeletedCells;
+import org.apache.hadoop.hbase.MemoryCompactionPolicy;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Store file writer does not do any compaction. Each cell written to either
the live or historical
+ * file. Regular (i.e., not-raw) scans that reads the latest put cells scans
only live files. To
+ * ensure the correctness of store file writer, we need to verify that live
files includes all live
+ * cells. This test indirectly verify this as follows. The test creates two
tables, each with one
+ * region and one store. The dual file writing (live vs historical) is
configured on only one of the
+ * tables. The test generates exact set of mutations on both tables. These
mutations include all
+ * types of cells and these cells are written to multiple files using multiple
memstore flushes.
+ * After writing all cells, the test first verify that both tables return the
same set of cells for
+ * regular and raw scans. Then the same verification is done after tables are
minor and finally
+ * major compacted. The test also verifies that flushes do not generate
historical files and the
+ * historical files are generated only when historical file generation is
enabled (by the config
+ * hbase.enable.historical.compaction.files).
+ */
+@Category({ MediumTests.class, RegionServerTests.class })
+@RunWith(Parameterized.class)
+public class TestStoreFileWriter {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestStoreFileWriter.class);
+ private final int ROW_NUM = 100;
+ private final Random RANDOM = new Random(11);
+ private final HBaseTestingUtil testUtil = new HBaseTestingUtil();
+ private HRegion[] regions = new HRegion[2];
+ private final byte[][] qualifiers =
+{ Bytes.toBytes("0"), Bytes.toBytes("1"), Bytes.toBytes("2") };
+ // This keeps track of all cells. It is a list of rows, each row is a list
of columns, each
+ // column is a list of CellInfo object
+ private ArrayList>> insertedCells;
+ private TableName[] tableName = new TableName[2];
+ private final Configuration conf = testUtil.getConfiguration();
+ private int flushCount = 0;
+
+ @Parameterized.Parameter(0)
+ public KeepDeletedCells keepDeletedCells;
+ @Parameterized.Parameter(1)
+ public int maxVersions;
+ @Parameterized.Parameter(2)
+ public boolean newVersionBehavior;
+
+
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2116510269 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 0m 5s | Docker failed to build run-specific yetus/hbase:tp-876}. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/34/console | | versions | git=2.25.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
virajjasani commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1604229161
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileWriter.java:
##
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder.NEW_VERSION_BEHAVIOR;
+import static
org.apache.hadoop.hbase.regionserver.StoreFileWriter.ENABLE_HISTORICAL_COMPACTION_FILES;
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Random;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.KeepDeletedCells;
+import org.apache.hadoop.hbase.MemoryCompactionPolicy;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Store file writer does not do any compaction. Each cell written to either
the live or historical
+ * file. Regular (i.e., not-raw) scans that reads the latest put cells scans
only live files. To
+ * ensure the correctness of store file writer, we need to verify that live
files includes all live
+ * cells. This test indirectly verify this as follows. The test creates two
tables, each with one
+ * region and one store. The dual file writing (live vs historical) is
configured on only one of the
+ * tables. The test generates exact set of mutations on both tables. These
mutations include all
+ * types of cells and these cells are written to multiple files using multiple
memstore flushes.
+ * After writing all cells, the test first verify that both tables return the
same set of cells for
+ * regular and raw scans. Then the same verification is done after tables are
minor and finally
+ * major compacted. The test also verifies that flushes do not generate
historical files and the
+ * historical files are generated only when historical file generation is
enabled (by the config
+ * hbase.enable.historical.compaction.files).
+ */
+@Category({ MediumTests.class, RegionServerTests.class })
+@RunWith(Parameterized.class)
+public class TestStoreFileWriter {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestStoreFileWriter.class);
+ private final int ROW_NUM = 100;
+ private final Random RANDOM = new Random(11);
+ private final HBaseTestingUtil testUtil = new HBaseTestingUtil();
+ private HRegion[] regions = new HRegion[2];
+ private final byte[][] qualifiers =
+{ Bytes.toBytes("0"), Bytes.toBytes("1"), Bytes.toBytes("2") };
+ // This keeps track of all cells. It is a list of rows, each row is a list
of columns, each
+ // column is a list of CellInfo object
+ private ArrayList>> insertedCells;
+ private TableName[] tableName = new TableName[2];
+ private final Configuration conf = testUtil.getConfiguration();
+ private int flushCount = 0;
+
+ @Parameterized.Parameter(0)
+ public KeepDeletedCells keepDeletedCells;
+ @Parameterized.Parameter(1)
+ public int maxVersions;
+ @Parameterized.Parameter(2)
+ public boolean newVersionBehavior;
+
+
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
virajjasani commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1604229161
##
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileWriter.java:
##
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder.NEW_VERSION_BEHAVIOR;
+import static
org.apache.hadoop.hbase.regionserver.StoreFileWriter.ENABLE_HISTORICAL_COMPACTION_FILES;
+import static org.junit.Assert.assertEquals;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Random;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtil;
+import org.apache.hadoop.hbase.KeepDeletedCells;
+import org.apache.hadoop.hbase.MemoryCompactionPolicy;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Delete;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.RegionServerTests;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Store file writer does not do any compaction. Each cell written to either
the live or historical
+ * file. Regular (i.e., not-raw) scans that reads the latest put cells scans
only live files. To
+ * ensure the correctness of store file writer, we need to verify that live
files includes all live
+ * cells. This test indirectly verify this as follows. The test creates two
tables, each with one
+ * region and one store. The dual file writing (live vs historical) is
configured on only one of the
+ * tables. The test generates exact set of mutations on both tables. These
mutations include all
+ * types of cells and these cells are written to multiple files using multiple
memstore flushes.
+ * After writing all cells, the test first verify that both tables return the
same set of cells for
+ * regular and raw scans. Then the same verification is done after tables are
minor and finally
+ * major compacted. The test also verifies that flushes do not generate
historical files and the
+ * historical files are generated only when historical file generation is
enabled (by the config
+ * hbase.enable.historical.compaction.files).
+ */
+@Category({ MediumTests.class, RegionServerTests.class })
+@RunWith(Parameterized.class)
+public class TestStoreFileWriter {
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestStoreFileWriter.class);
+ private final int ROW_NUM = 100;
+ private final Random RANDOM = new Random(11);
+ private final HBaseTestingUtil testUtil = new HBaseTestingUtil();
+ private HRegion[] regions = new HRegion[2];
+ private final byte[][] qualifiers =
+{ Bytes.toBytes("0"), Bytes.toBytes("1"), Bytes.toBytes("2") };
+ // This keeps track of all cells. It is a list of rows, each row is a list
of columns, each
+ // column is a list of CellInfo object
+ private ArrayList>> insertedCells;
+ private TableName[] tableName = new TableName[2];
+ private final Configuration conf = testUtil.getConfiguration();
+ private int flushCount = 0;
+
+ @Parameterized.Parameter(0)
+ public KeepDeletedCells keepDeletedCells;
+ @Parameterized.Parameter(1)
+ public int maxVersions;
+ @Parameterized.Parameter(2)
+ public boolean newVersionBehavior;
+
+
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2115317652 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 32s | master passed | | +1 :green_heart: | compile | 1m 28s | master passed | | +1 :green_heart: | shadedjars | 7m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | master passed | | -0 :warning: | patch | 8m 28s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 1m 25s | the patch passed | | +1 :green_heart: | javac | 1m 25s | the patch passed | | +1 :green_heart: | shadedjars | 7m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 293m 35s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 20m 42s | hbase-mapreduce in the patch passed. | | | | 346m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/33/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux aed95bfce225 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 716adf50e9 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/33/testReport/ | | Max. process+thread count | 4880 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/33/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2115235512 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | compile | 1m 48s | master passed | | +1 :green_heart: | shadedjars | 6m 52s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 1s | master passed | | -0 :warning: | patch | 8m 14s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 33s | the patch passed | | +1 :green_heart: | compile | 1m 25s | the patch passed | | +1 :green_heart: | javac | 1m 25s | the patch passed | | +1 :green_heart: | shadedjars | 7m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 263m 9s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 13m 38s | hbase-mapreduce in the patch passed. | | | | 309m 51s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/33/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8a6f4107cf90 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 716adf50e9 | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/33/testReport/ | | Max. process+thread count | 4532 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/33/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2115125977 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 14s | master passed | | +1 :green_heart: | compile | 1m 7s | master passed | | +1 :green_heart: | shadedjars | 5m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | | -0 :warning: | patch | 6m 36s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 48s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 5m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 222m 5s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 15m 1s | hbase-mapreduce in the patch passed. | | | | 263m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/33/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 836ac80252d7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 716adf50e9 | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/33/testReport/ | | Max. process+thread count | 5363 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/33/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2114510060 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 55s | master passed | | +1 :green_heart: | compile | 3m 12s | master passed | | +1 :green_heart: | checkstyle | 0m 52s | master passed | | +1 :green_heart: | spotless | 0m 45s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 3s | master passed | | -0 :warning: | patch | 1m 43s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 52s | the patch passed | | +1 :green_heart: | compile | 3m 11s | the patch passed | | +1 :green_heart: | javac | 3m 11s | the patch passed | | -0 :warning: | checkstyle | 0m 37s | hbase-server: The patch generated 1 new + 46 unchanged - 0 fixed = 47 total (was 46) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 4m 57s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 42s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 20s | The patch does not generate ASF License warnings. | | | | 32m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/33/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 1bf9fde0dcfe 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 716adf50e9 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/33/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/33/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2114088475 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 3s | https://github.com/apache/hbase/pull/5545 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/32/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2114088119 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 3s | https://github.com/apache/hbase/pull/5545 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/32/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2114088089 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 3s | https://github.com/apache/hbase/pull/5545 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/32/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2114087829 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 3s | https://github.com/apache/hbase/pull/5545 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/32/console | | versions | git=2.25.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1602580409
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java:
##
@@ -50,15 +50,15 @@ public interface StoreFileManager {
*/
@RestrictedApi(explanation = "Should only be called in StoreEngine", link =
"",
allowedOnPath =
".*(/org/apache/hadoop/hbase/regionserver/StoreEngine.java|/src/test/.*)")
- void loadFiles(List storeFiles);
+ void loadFiles(List storeFiles) throws IOException;
Review Comment:
Anyway, not a blocker. Can debug later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2113965661 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 3s | https://github.com/apache/hbase/pull/5545 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/31/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2113965652 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 3s | https://github.com/apache/hbase/pull/5545 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/31/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2113965649 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 2s | https://github.com/apache/hbase/pull/5545 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/31/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2113965547 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 3s | https://github.com/apache/hbase/pull/5545 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/31/console | | versions | git=2.25.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1602564157
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java:
##
@@ -50,15 +50,15 @@ public interface StoreFileManager {
*/
@RestrictedApi(explanation = "Should only be called in StoreEngine", link =
"",
allowedOnPath =
".*(/org/apache/hadoop/hbase/regionserver/StoreEngine.java|/src/test/.*)")
- void loadFiles(List storeFiles);
+ void loadFiles(List storeFiles) throws IOException;
Review Comment:
I do not know the answer. I was going to debug and find the answer. However,
I could not find a test that calls this method with non-empty storeFiles. I
need to device a test where I need to generate some files and then close the
store, and reopen it while debugging it. I will try to do it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1602560727
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java:
##
@@ -48,17 +52,38 @@ class DefaultStoreFileManager implements StoreFileManager {
private final CompactionConfiguration comConf;
private final int blockingFileCount;
private final Comparator storeFileComparator;
- /**
- * List of store files inside this store. This is an immutable list that is
atomically replaced
- * when its contents change.
- */
- private volatile ImmutableList storefiles = ImmutableList.of();
+
+ static class StoreFileList {
+/**
+ * List of store files inside this store. This is an immutable list that
is atomically replaced
+ * when its contents change.
+ */
+final ImmutableList all;
+/**
+ * List of store files that include the latest cells inside this store.
This is an immutable
+ * list that is atomically replaced when its contents change.
+ */
+@Nullable
+final ImmutableList live;
+
+StoreFileList(ImmutableList storeFiles,
ImmutableList liveStoreFiles) {
+ this.all = storeFiles;
+ this.live = liveStoreFiles;
+}
+ }
+
+ private static final StoreFileList EMPTY_STORE_FILE_LIST =
+new StoreFileList(ImmutableList.of(), null);
+
+ private volatile StoreFileList storeFiles = EMPTY_STORE_FILE_LIST;
Review Comment:
Got it. I changed the code such that null is passed only when live tracking
is disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1601087841
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java:
##
@@ -48,17 +52,38 @@ class DefaultStoreFileManager implements StoreFileManager {
private final CompactionConfiguration comConf;
private final int blockingFileCount;
private final Comparator storeFileComparator;
- /**
- * List of store files inside this store. This is an immutable list that is
atomically replaced
- * when its contents change.
- */
- private volatile ImmutableList storefiles = ImmutableList.of();
+
+ static class StoreFileList {
+/**
+ * List of store files inside this store. This is an immutable list that
is atomically replaced
+ * when its contents change.
+ */
+final ImmutableList all;
+/**
+ * List of store files that include the latest cells inside this store.
This is an immutable
+ * list that is atomically replaced when its contents change.
+ */
+@Nullable
+final ImmutableList live;
+
+StoreFileList(ImmutableList storeFiles,
ImmutableList liveStoreFiles) {
+ this.all = storeFiles;
+ this.live = liveStoreFiles;
+}
+ }
+
+ private static final StoreFileList EMPTY_STORE_FILE_LIST =
+new StoreFileList(ImmutableList.of(), null);
+
+ private volatile StoreFileList storeFiles = EMPTY_STORE_FILE_LIST;
Review Comment:
When initializating we do not need to test whether live file tracking is
enabled? I see in later code, if live file tracking is enabled we will pass
ImmutableList.of() instead of null here.
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java:
##
@@ -50,15 +50,15 @@ public interface StoreFileManager {
*/
@RestrictedApi(explanation = "Should only be called in StoreEngine", link =
"",
allowedOnPath =
".*(/org/apache/hadoop/hbase/regionserver/StoreEngine.java|/src/test/.*)")
- void loadFiles(List storeFiles);
+ void loadFiles(List storeFiles) throws IOException;
Review Comment:
I know the there are throws declarations, what I mean here is whether it
will actually throw any exceptions out. For example, in loadFiles, maybe we
have already called initReader for all files so there is no problem? Just want
to confirm this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
apurtell commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2111577285 @Apache9 I think there was only one issue considered a blocker and @kadirozde has addressed it. There are three failing tests in the precommit but all are related to quotas so are unlikely due to this. I would like to merge this interesting change and test it further if there are no objections. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2088988994 @Apache9, would you please review the latest commit and let me know if you have more comments for this PR? I also updated the design doc to reflect the changes based on your review comments. Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2075973876 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 6s | master passed | | +1 :green_heart: | compile | 1m 19s | master passed | | +1 :green_heart: | shadedjars | 6m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | master passed | | -0 :warning: | patch | 8m 3s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 51s | the patch passed | | +1 :green_heart: | compile | 1m 25s | the patch passed | | +1 :green_heart: | javac | 1m 25s | the patch passed | | +1 :green_heart: | shadedjars | 6m 29s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 277m 55s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 15m 9s | hbase-mapreduce in the patch passed. | | | | 325m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/30/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9e27f690482d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c6e776eea | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/30/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/30/testReport/ | | Max. process+thread count | 4705 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/30/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2075919947 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 36s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 5m 9s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | | -0 :warning: | patch | 6m 7s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 33s | the patch passed | | +1 :green_heart: | compile | 1m 3s | the patch passed | | +1 :green_heart: | javac | 1m 3s | the patch passed | | +1 :green_heart: | shadedjars | 5m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 240m 30s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 15m 21s | hbase-mapreduce in the patch passed. | | | | 280m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/30/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9c186612914f 5.4.0-176-generic #196-Ubuntu SMP Fri Mar 22 16:46:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c6e776eea | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/30/testReport/ | | Max. process+thread count | 5077 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/30/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2075904068 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 15s | master passed | | +1 :green_heart: | compile | 1m 13s | master passed | | +1 :green_heart: | shadedjars | 5m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | master passed | | -0 :warning: | patch | 6m 41s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 59s | the patch passed | | +1 :green_heart: | compile | 1m 14s | the patch passed | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | shadedjars | 5m 43s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 224m 58s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 14m 27s | hbase-mapreduce in the patch passed. | | | | 266m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/30/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 952f4e74a2a6 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c6e776eea | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/30/testReport/ | | Max. process+thread count | 4468 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/30/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2075520201 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 9s | master passed | | +1 :green_heart: | compile | 3m 0s | master passed | | +1 :green_heart: | checkstyle | 0m 48s | master passed | | +1 :green_heart: | spotless | 0m 41s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 54s | master passed | | -0 :warning: | patch | 0m 38s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | compile | 2m 59s | the patch passed | | +1 :green_heart: | javac | 2m 59s | the patch passed | | -0 :warning: | checkstyle | 0m 35s | hbase-server: The patch generated 1 new + 46 unchanged - 0 fixed = 47 total (was 46) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 9s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 46s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 58s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 33m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/30/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 3b9f5a04bc13 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 6c6e776eea | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/30/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 82 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/30/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2074180292 > There is only one blocker, about the concurrency control. We'd better still hold the consistent view while getting storefile list. > > The other is not a blocker but still better to make it clear that whether we will throw any exception for some storefile management operations. > > Thanks. @Apache9, I made the requested changes for having consistent view while getting store files. I also took the liberty to rename storefiles to storeFiles and getStorefiles() to getStoreFiles(). The hfile reader throws IOException. DefaultStoreFileManager reads the file metadata during loading the files to see which of the loaded files are live (since DefaultStoreFileManager maintains a separate list for live files). Therefore, I had to change the method signature. Thank you for review again! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2074177990 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 3s | https://github.com/apache/hbase/pull/5545 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/29/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2074177955 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 3s | https://github.com/apache/hbase/pull/5545 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/29/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2074177958 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 2s | https://github.com/apache/hbase/pull/5545 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/29/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2074177864 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 4s | https://github.com/apache/hbase/pull/5545 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/29/console | | versions | git=2.25.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1577321980
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java:
##
@@ -86,13 +111,20 @@ public Collection getCompactedfiles() {
}
@Override
- public void insertNewFiles(Collection sfs) {
+ public void insertNewFiles(Collection sfs) throws IOException {
+if (enableLiveFileTracking) {
+ this.liveStoreFiles =
ImmutableList.sortedCopyOf(getStoreFileComparator(),
Review Comment:
Sounds good! I will make the changes as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1577320047
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java:
##
@@ -50,15 +50,15 @@ public interface StoreFileManager {
*/
@RestrictedApi(explanation = "Should only be called in StoreEngine", link =
"",
allowedOnPath =
".*(/org/apache/hadoop/hbase/regionserver/StoreEngine.java|/src/test/.*)")
- void loadFiles(List storeFiles);
+ void loadFiles(List storeFiles) throws IOException;
Review Comment:
Sorry I missed this one before. When DefaultStoreFileManager loads the store
files, it also reads the file metadata to see if they are live or not in order
to form the list of live store files. Reading file can generate IOException.
Please see
```
private List getLiveFiles(Collection storeFiles)
throws IOException {
List liveFiles = new ArrayList<>(storeFiles.size());
for (HStoreFile file : storeFiles) {
file.initReader();
if (!file.isHistorical()) {
liveFiles.add(file);
}
}
return liveFiles;
}
@Override
public void loadFiles(List storeFiles) throws IOException {
this.storeFiles = new
StoreFileList(ImmutableList.sortedCopyOf(storeFileComparator, storeFiles),
enableLiveFileTracking
? ImmutableList.sortedCopyOf(storeFileComparator,
getLiveFiles(storeFiles))
: null);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1577104587
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java:
##
@@ -50,15 +50,15 @@ public interface StoreFileManager {
*/
@RestrictedApi(explanation = "Should only be called in StoreEngine", link =
"",
allowedOnPath =
".*(/org/apache/hadoop/hbase/regionserver/StoreEngine.java|/src/test/.*)")
- void loadFiles(List storeFiles);
+ void loadFiles(List storeFiles) throws IOException;
Review Comment:
And also this one, could you please verify whether we actually throw any
exceptions here? We'd better not declare throws if there is no actual exception
thrown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1577103908
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java:
##
@@ -86,13 +111,20 @@ public Collection getCompactedfiles() {
}
@Override
- public void insertNewFiles(Collection sfs) {
+ public void insertNewFiles(Collection sfs) throws IOException {
+if (enableLiveFileTracking) {
+ this.liveStoreFiles =
ImmutableList.sortedCopyOf(getStoreFileComparator(),
Review Comment:
I suggest that, we introduce a something like a Pair for storing all
storefiles and live storefiles in a single field, and mark it as volatile, so
the reader can get always get a consistent view. Something like
```
class StoreFileList {
@Nonnull
final ImmutableList storefiles;
@Nullable
final ImmutableList liveStorefiles;
StoreFileList(ImmutableList storefiles,
ImmutableList liveStorefiles) {
this.storefiles = storefiles;
this.liveStorefiles = liveStorefiles;
}
}
private static final StoreFileList EMPTY_STOREFILE_LIST = new
StoreFileList(ImmutableList.of(), null);
private volatile StoreFileList storefiles = EMPTY_STOREFILE_LIST;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2073830963 Ah, sorry, forgot this one... Will take a look soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
bbeaudreault commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2073825447 @Apache9 Can you respond here? I think we should move towards merging this, @kadirozde has been very accommodating. I also want to make sure you're happy with it, since your perspective is insightful and greatly appreciated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2067815560 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | compile | 1m 41s | master passed | | +1 :green_heart: | shadedjars | 8m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 4s | master passed | | -0 :warning: | patch | 9m 40s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 17s | the patch passed | | +1 :green_heart: | compile | 1m 43s | the patch passed | | +1 :green_heart: | javac | 1m 43s | the patch passed | | +1 :green_heart: | shadedjars | 7m 45s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 1s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 241m 18s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 14m 49s | hbase-mapreduce in the patch passed. | | | | 291m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/28/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d281983b0048 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3539581268 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/28/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/28/testReport/ | | Max. process+thread count | 5569 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/28/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2067813556 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 44s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 10s | master passed | | +1 :green_heart: | compile | 1m 18s | master passed | | +1 :green_heart: | shadedjars | 5m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | | -0 :warning: | patch | 6m 48s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 26s | the patch passed | | +1 :green_heart: | compile | 1m 0s | the patch passed | | +1 :green_heart: | javac | 1m 0s | the patch passed | | +1 :green_heart: | shadedjars | 5m 9s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 236m 35s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 15m 8s | hbase-mapreduce in the patch passed. | | | | 278m 20s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/28/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8464e44b0505 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3539581268 | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/28/testReport/ | | Max. process+thread count | 5716 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/28/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2067809359 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 37s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 19s | master passed | | +1 :green_heart: | compile | 1m 14s | master passed | | +1 :green_heart: | shadedjars | 5m 46s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | | -0 :warning: | patch | 6m 43s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 58s | the patch passed | | +1 :green_heart: | compile | 1m 14s | the patch passed | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | shadedjars | 5m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 211m 46s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 13m 30s | hbase-mapreduce in the patch passed. | | | | 252m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/28/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 074e7e460374 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3539581268 | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/28/testReport/ | | Max. process+thread count | 5135 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/28/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2067761930 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 7s | master passed | | +1 :green_heart: | compile | 2m 58s | master passed | | +1 :green_heart: | checkstyle | 0m 46s | master passed | | +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 57s | master passed | | -0 :warning: | patch | 0m 39s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 42s | the patch passed | | +1 :green_heart: | compile | 3m 20s | the patch passed | | +1 :green_heart: | javac | 3m 20s | the patch passed | | -0 :warning: | checkstyle | 0m 45s | hbase-server: The patch generated 1 new + 39 unchanged - 0 fixed = 40 total (was 39) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 49s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 50s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 33m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/28/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux b497492f055f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 3539581268 | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/28/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 81 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/28/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1559993483
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java:
##
@@ -86,13 +111,20 @@ public Collection getCompactedfiles() {
}
@Override
- public void insertNewFiles(Collection sfs) {
+ public void insertNewFiles(Collection sfs) throws IOException {
+if (enableLiveFileTracking) {
+ this.liveStoreFiles =
ImmutableList.sortedCopyOf(getStoreFileComparator(),
Review Comment:
I just want to make sure that we are on the same page here. You think we
need to make this method atomic even though
(1) The class is not thread-safe as specified here
`Default implementation of StoreFileManager. Not thread-safe.`
(2) There is only one caller which is StoreEngine and it takes a write lock
whenever it calls a StoreFileManager method, for example
```
public void addStoreFiles(Collection storeFiles,
IOExceptionRunnable actionAfterAdding) throws IOException {
storeFileTracker.add(StoreUtils.toStoreFileInfo(storeFiles));
writeLock();
try {
storeFileManager.insertNewFiles(storeFiles);
actionAfterAdding.run();
} finally {
// We need the lock, as long as we are updating the storeFiles
// or changing the memstore. Let us release it before calling
// notifyChangeReadersObservers. See HBASE-4485 for a possible
// deadlock scenario that could have happened if continue to hold
// the lock.
writeUnlock();
}
}
```
I can easily add a write lock here and to the other methods but given the
above points, is not it redundant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2048360084 > For me the only big problem is how to deal with the updating of store file list. Please check my response on this. If you still want me to add a lock to protect the store file list updates, I will do it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1559993483
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java:
##
@@ -86,13 +111,20 @@ public Collection getCompactedfiles() {
}
@Override
- public void insertNewFiles(Collection sfs) {
+ public void insertNewFiles(Collection sfs) throws IOException {
+if (enableLiveFileTracking) {
+ this.liveStoreFiles =
ImmutableList.sortedCopyOf(getStoreFileComparator(),
Review Comment:
I just want to make sure that we are on the same page here. You think we
need to make this method atomic even though
(1) The class is not thread-safe as specified here
`Default implementation of StoreFileManager. Not thread-safe.`
(2) There is only one caller which is StoreEngine and it takes a write lock
whenever it calls a StoreFileManager methods, for example
```
public void addStoreFiles(Collection storeFiles,
IOExceptionRunnable actionAfterAdding) throws IOException {
storeFileTracker.add(StoreUtils.toStoreFileInfo(storeFiles));
writeLock();
try {
storeFileManager.insertNewFiles(storeFiles);
actionAfterAdding.run();
} finally {
// We need the lock, as long as we are updating the storeFiles
// or changing the memstore. Let us release it before calling
// notifyChangeReadersObservers. See HBASE-4485 for a possible
// deadlock scenario that could have happened if continue to hold
// the lock.
writeUnlock();
}
}
```
I can easily add a write lock here and to the other methods but given the
above points, is not it redundant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1559993483
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java:
##
@@ -86,13 +111,20 @@ public Collection getCompactedfiles() {
}
@Override
- public void insertNewFiles(Collection sfs) {
+ public void insertNewFiles(Collection sfs) throws IOException {
+if (enableLiveFileTracking) {
+ this.liveStoreFiles =
ImmutableList.sortedCopyOf(getStoreFileComparator(),
Review Comment:
I just want to make sure that we are on the same page here. You think we
need to make this method atomic even though
(1) The class is not thread-safe as specified here
`/**
* Default implementation of StoreFileManager. Not thread-safe.
*/`
(2) There is only one caller which is StoreEngine and it takes a write lock
whenever it calls a StoreFileManager methods, for example
` /**
* Add the store files to store file manager, and also record it in the
store file tracker.
*
* The {@code actionAfterAdding} will be executed after the insertion to
store file manager, under
* the lock protection. Usually this is for clear the memstore snapshot.
*/
public void addStoreFiles(Collection storeFiles,
IOExceptionRunnable actionAfterAdding) throws IOException {
storeFileTracker.add(StoreUtils.toStoreFileInfo(storeFiles));
writeLock();
try {
storeFileManager.insertNewFiles(storeFiles);
actionAfterAdding.run();
} finally {
// We need the lock, as long as we are updating the storeFiles
// or changing the memstore. Let us release it before calling
// notifyChangeReadersObservers. See HBASE-4485 for a possible
// deadlock scenario that could have happened if continue to hold
// the lock.
writeUnlock();
}
}`
I can easily add a write lock here and to the other methods but given the
above points, is not it redundant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1558971974
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -256,156 +234,571 @@ public void appendMobMetadata(SetMultimap mobRefSet) throws I
public void appendTrackedTimestampsToMetadata() throws IOException {
// TODO: The StoreFileReader always converts the byte[] to TimeRange
// via TimeRangeTracker, so we should write the serialization data of
TimeRange directly.
-appendFileInfo(TIMERANGE_KEY,
TimeRangeTracker.toByteArray(timeRangeTracker));
-appendFileInfo(EARLIEST_PUT_TS, Bytes.toBytes(earliestPutTs));
+liveFileWriter.appendTrackedTimestampsToMetadata();
Review Comment:
Checked the code, besides calling the (old) StoreFileWriter, this method is
only called in HFileOutputFormat2, for appending metadata for store files
generated by MR job. Seems better to introduce a special method in
StoreFileWriter for it. Anyway, not a problem for this PR. Can file another
issue for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1558965605
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -256,156 +234,571 @@ public void appendMobMetadata(SetMultimap mobRefSet) throws I
public void appendTrackedTimestampsToMetadata() throws IOException {
// TODO: The StoreFileReader always converts the byte[] to TimeRange
// via TimeRangeTracker, so we should write the serialization data of
TimeRange directly.
-appendFileInfo(TIMERANGE_KEY,
TimeRangeTracker.toByteArray(timeRangeTracker));
-appendFileInfo(EARLIEST_PUT_TS, Bytes.toBytes(earliestPutTs));
+liveFileWriter.appendTrackedTimestampsToMetadata();
+if (historicalFileWriter != null) {
+ historicalFileWriter.appendTrackedTimestampsToMetadata();
+}
}
/**
* Record the earlest Put timestamp. If the timeRangeTracker is not set,
update TimeRangeTracker
* to include the timestamp of this key
*/
public void trackTimestamps(final Cell cell) {
-if (KeyValue.Type.Put.getCode() == cell.getTypeByte()) {
- earliestPutTs = Math.min(earliestPutTs, cell.getTimestamp());
+liveFileWriter.trackTimestamps(cell);
Review Comment:
Checked the old code, this method is only called in StoreFileWriter.append,
so I think you can just remove this method here, only keep it in
SIngleStoreFileWriter, and make it private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1558961072
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -181,7 +197,10 @@ public long getPos() throws IOException {
*/
public void appendMetadata(final long maxSequenceId, final boolean
majorCompaction)
throws IOException {
-appendMetadata(maxSequenceId, majorCompaction, Collections.emptySet());
+liveFileWriter.appendMetadata(maxSequenceId, majorCompaction);
Review Comment:
OK, checked the other multi file writer implementation, found HBASE-15400,
where we explained why we use the same maxSequenceId for multiple genedated
store files. So at least this is not a critical problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1558933042
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStoreFile.java:
##
@@ -138,6 +140,12 @@ public class HStoreFile implements StoreFile {
// Indicates if the file got compacted
private volatile boolean compactedAway = false;
+ // Indicate if the file contains historical cell versions. This is used when
+ // hbase.enable.historical.compaction.files is set to true. In that case,
compactions
+ // can generate two files, one with the live cell versions and the other
with the remaining
+ // (historical) cell versions.
+ private volatile boolean isHistorical = false;
Review Comment:
The new name seems good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1558933426
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java:
##
@@ -50,15 +50,15 @@ public interface StoreFileManager {
*/
@RestrictedApi(explanation = "Should only be called in StoreEngine", link =
"",
allowedOnPath =
".*(/org/apache/hadoop/hbase/regionserver/StoreEngine.java|/src/test/.*)")
- void loadFiles(List storeFiles);
+ void loadFiles(List storeFiles) throws IOException;
Review Comment:
Ping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1558932155
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java:
##
@@ -86,13 +111,20 @@ public Collection getCompactedfiles() {
}
@Override
- public void insertNewFiles(Collection sfs) {
+ public void insertNewFiles(Collection sfs) throws IOException {
+if (enableLiveFileTracking) {
+ this.liveStoreFiles =
ImmutableList.sortedCopyOf(getStoreFileComparator(),
Review Comment:
The point here is that, in the past, we only have a single list for storing
the store files, so in the reader's view, there is no inconsitency problem. But
now, we have two lists for storing store files, and the update is not atomic,
what if the reader gets the list in the middle of the updates to these two
fields?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545: URL: https://github.com/apache/hbase/pull/5545#discussion_r1558928060 ## hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java: ## @@ -464,7 +464,6 @@ protected boolean performCompaction(FileDetails fd, InternalScanner scanner, Cel lastCleanCell = null; lastCleanCellSeqId = 0; } - writer.append(c); Review Comment: Let's not mix different things up? I'm not sure if there are other side effects about this change... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2045810903 @Apache9, @bbeaudreault, @virajjasani, @apurtell, please let me know if there is any review item left to be addressed. From my perspective, I have addressed all review comments. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545925927
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -247,7 +222,10 @@ public void appendMetadata(final long maxSequenceId, final
boolean majorCompacti
* @throws IOException problem writing to FS
*/
public void appendMobMetadata(SetMultimap mobRefSet)
throws IOException {
-writer.appendFileInfo(MOB_FILE_REFS,
MobUtils.serializeMobFileRefs(mobRefSet));
+liveFileWriter.appendMobMetadata(mobRefSet);
Review Comment:
I have done that in the last commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545925927
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -247,7 +222,10 @@ public void appendMetadata(final long maxSequenceId, final
boolean majorCompacti
* @throws IOException problem writing to FS
*/
public void appendMobMetadata(SetMultimap mobRefSet)
throws IOException {
-writer.appendFileInfo(MOB_FILE_REFS,
MobUtils.serializeMobFileRefs(mobRefSet));
+liveFileWriter.appendMobMetadata(mobRefSet);
Review Comment:
I have done that the last commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2028948735 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 27s | master passed | | +1 :green_heart: | compile | 1m 0s | master passed | | +1 :green_heart: | shadedjars | 5m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 53s | master passed | | -0 :warning: | patch | 6m 22s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 12s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 5m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 245m 4s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 14m 52s | hbase-mapreduce in the patch passed. | | | | 285m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/27/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 344c7def2d2c 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9adca10e9c | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/27/testReport/ | | Max. process+thread count | 5958 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/27/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2028942924 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 7s | master passed | | +1 :green_heart: | compile | 1m 5s | master passed | | +1 :green_heart: | shadedjars | 5m 51s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | | -0 :warning: | patch | 6m 42s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 1s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | shadedjars | 5m 51s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 221m 3s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 14m 19s | hbase-mapreduce in the patch passed. | | | | 262m 3s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/27/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 94b30f1ef591 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9adca10e9c | | Default Java | Eclipse Adoptium-11.0.17+8 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/27/testReport/ | | Max. process+thread count | 5713 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/27/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2028937969 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 5s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 39s | master passed | | +1 :green_heart: | compile | 1m 6s | master passed | | +1 :green_heart: | shadedjars | 5m 21s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | | -0 :warning: | patch | 6m 11s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 51s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | shadedjars | 5m 19s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 202m 54s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 12m 52s | hbase-mapreduce in the patch passed. | | | | 240m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/27/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e40ab7e3a51f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9adca10e9c | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/27/testReport/ | | Max. process+thread count | 5447 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/27/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
bbeaudreault commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545796726
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java:
##
@@ -86,13 +111,20 @@ public Collection getCompactedfiles() {
}
@Override
- public void insertNewFiles(Collection sfs) {
+ public void insertNewFiles(Collection sfs) throws IOException {
+if (enableLiveFileTracking) {
+ this.liveStoreFiles =
ImmutableList.sortedCopyOf(getStoreFileComparator(),
Review Comment:
Could you check if mob is enabled and, if so, log a warn and ignore the dual
writer config value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
bbeaudreault commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545798434
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -247,7 +222,10 @@ public void appendMetadata(final long maxSequenceId, final
boolean majorCompacti
* @throws IOException problem writing to FS
*/
public void appendMobMetadata(SetMultimap mobRefSet)
throws IOException {
-writer.appendFileInfo(MOB_FILE_REFS,
MobUtils.serializeMobFileRefs(mobRefSet));
+liveFileWriter.appendMobMetadata(mobRefSet);
Review Comment:
Could you check if mob is enabled and, if so, log a warn and ignore the dual
writer config value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
bbeaudreault commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545796726
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java:
##
@@ -86,13 +111,20 @@ public Collection getCompactedfiles() {
}
@Override
- public void insertNewFiles(Collection sfs) {
+ public void insertNewFiles(Collection sfs) throws IOException {
+if (enableLiveFileTracking) {
+ this.liveStoreFiles =
ImmutableList.sortedCopyOf(getStoreFileComparator(),
Review Comment:
Could you check if mob is enabled and, if so, log a warn and ignore the dual
writer config value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2028883478 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 6s | master passed | | +1 :green_heart: | compile | 2m 56s | master passed | | +1 :green_heart: | checkstyle | 0m 45s | master passed | | +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 52s | master passed | | -0 :warning: | patch | 0m 39s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 49s | the patch passed | | +1 :green_heart: | compile | 3m 39s | the patch passed | | +1 :green_heart: | javac | 3m 39s | the patch passed | | -0 :warning: | checkstyle | 0m 39s | hbase-server: The patch generated 1 new + 39 unchanged - 0 fixed = 40 total (was 39) | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 39s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 35m 50s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/27/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux e2a7ab773db0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9adca10e9c | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/27/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 80 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/27/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545770119
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -247,7 +222,10 @@ public void appendMetadata(final long maxSequenceId, final
boolean majorCompacti
* @throws IOException problem writing to FS
*/
public void appendMobMetadata(SetMultimap mobRefSet)
throws IOException {
-writer.appendFileInfo(MOB_FILE_REFS,
MobUtils.serializeMobFileRefs(mobRefSet));
+liveFileWriter.appendMobMetadata(mobRefSet);
Review Comment:
Since we moved dual file writing into StoreFileWriter and all compaction
methods use StoreFileWriter, there is no way to prevent it. Please note by
default dual file writing is disabled, one needs to enable it to use it by
setting hbase.enable.historical.compaction.files to true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545755759
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -256,156 +234,571 @@ public void appendMobMetadata(SetMultimap mobRefSet) throws I
public void appendTrackedTimestampsToMetadata() throws IOException {
// TODO: The StoreFileReader always converts the byte[] to TimeRange
// via TimeRangeTracker, so we should write the serialization data of
TimeRange directly.
-appendFileInfo(TIMERANGE_KEY,
TimeRangeTracker.toByteArray(timeRangeTracker));
-appendFileInfo(EARLIEST_PUT_TS, Bytes.toBytes(earliestPutTs));
+liveFileWriter.appendTrackedTimestampsToMetadata();
+if (historicalFileWriter != null) {
+ historicalFileWriter.appendTrackedTimestampsToMetadata();
+}
}
/**
* Record the earlest Put timestamp. If the timeRangeTracker is not set,
update TimeRangeTracker
* to include the timestamp of this key
*/
public void trackTimestamps(final Cell cell) {
-if (KeyValue.Type.Put.getCode() == cell.getTypeByte()) {
- earliestPutTs = Math.min(earliestPutTs, cell.getTimestamp());
+liveFileWriter.trackTimestamps(cell);
Review Comment:
The delete code here is moved to SingleStoreFileWriter#trackTimestamps(). So
the behavior did not change.
`private void trackTimestamps(final Cell cell) {
if (KeyValue.Type.Put.getCode() == cell.getTypeByte()) {
earliestPutTs = Math.min(earliestPutTs, cell.getTimestamp());
}
timeRangeTracker.includeTimestamp(cell);
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545755759
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -256,156 +234,571 @@ public void appendMobMetadata(SetMultimap mobRefSet) throws I
public void appendTrackedTimestampsToMetadata() throws IOException {
// TODO: The StoreFileReader always converts the byte[] to TimeRange
// via TimeRangeTracker, so we should write the serialization data of
TimeRange directly.
-appendFileInfo(TIMERANGE_KEY,
TimeRangeTracker.toByteArray(timeRangeTracker));
-appendFileInfo(EARLIEST_PUT_TS, Bytes.toBytes(earliestPutTs));
+liveFileWriter.appendTrackedTimestampsToMetadata();
+if (historicalFileWriter != null) {
+ historicalFileWriter.appendTrackedTimestampsToMetadata();
+}
}
/**
* Record the earlest Put timestamp. If the timeRangeTracker is not set,
update TimeRangeTracker
* to include the timestamp of this key
*/
public void trackTimestamps(final Cell cell) {
-if (KeyValue.Type.Put.getCode() == cell.getTypeByte()) {
- earliestPutTs = Math.min(earliestPutTs, cell.getTimestamp());
+liveFileWriter.trackTimestamps(cell);
Review Comment:
The delete code here is move to SingleStoreFileWriter#trackTimestamps(). So
the behavior did not change.
`private void trackTimestamps(final Cell cell) {
if (KeyValue.Type.Put.getCode() == cell.getTypeByte()) {
earliestPutTs = Math.min(earliestPutTs, cell.getTimestamp());
}
timeRangeTracker.includeTimestamp(cell);
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545554451
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -105,72 +132,44 @@ public class StoreFileWriter implements CellSink,
ShipperListener {
* @param fileContextThe HFile context
* @param shouldDropCacheBehind Drop pages written to page cache after
writing the store file.
* @param compactedFilesSupplier Returns the {@link HStore} compacted files
which not archived
+ * @param comparator Cell comparator
+ * @param maxVersionsmax cell versions
+ * @param newVersionBehavior enable new version behavior
* @throws IOException problem writing to FS
*/
- private StoreFileWriter(FileSystem fs, Path path, final Configuration conf,
CacheConfig cacheConf,
-BloomType bloomType, long maxKeys, InetSocketAddress[] favoredNodes,
HFileContext fileContext,
-boolean shouldDropCacheBehind, Supplier>
compactedFilesSupplier)
-throws IOException {
+ private StoreFileWriter(FileSystem fs, Path liveFilePath, Path
historicalFilePath,
+final Configuration conf, CacheConfig cacheConf, BloomType bloomType, long
maxKeys,
+InetSocketAddress[] favoredNodes, HFileContext fileContext, boolean
shouldDropCacheBehind,
+Supplier> compactedFilesSupplier, CellComparator
comparator,
+int maxVersions, boolean newVersionBehavior) throws IOException {
+this.fs = fs;
+this.historicalFilePath = historicalFilePath;
+this.conf = conf;
+this.cacheConf = cacheConf;
+this.bloomType = bloomType;
+this.maxKeys = maxKeys;
+this.favoredNodes = favoredNodes;
+this.fileContext = fileContext;
+this.shouldDropCacheBehind = shouldDropCacheBehind;
this.compactedFilesSupplier = compactedFilesSupplier;
-this.timeRangeTracker =
TimeRangeTracker.create(TimeRangeTracker.Type.NON_SYNC);
-// TODO : Change all writers to be specifically created for compaction
context
-writer =
- HFile.getWriterFactory(conf, cacheConf).withPath(fs,
path).withFavoredNodes(favoredNodes)
-
.withFileContext(fileContext).withShouldDropCacheBehind(shouldDropCacheBehind).create();
-
-generalBloomFilterWriter =
BloomFilterFactory.createGeneralBloomAtWrite(conf, cacheConf,
- bloomType, (int) Math.min(maxKeys, Integer.MAX_VALUE), writer);
-
-if (generalBloomFilterWriter != null) {
- this.bloomType = bloomType;
- this.bloomParam = BloomFilterUtil.getBloomFilterParam(bloomType, conf);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Bloom filter type for " + path + ": " + this.bloomType + ",
param: "
- + (bloomType == BloomType.ROWPREFIX_FIXED_LENGTH
-? Bytes.toInt(bloomParam)
-: Bytes.toStringBinary(bloomParam))
- + ", " + generalBloomFilterWriter.getClass().getSimpleName());
- }
- // init bloom context
- switch (bloomType) {
-case ROW:
- bloomContext =
-new RowBloomContext(generalBloomFilterWriter,
fileContext.getCellComparator());
- break;
-case ROWCOL:
- bloomContext =
-new RowColBloomContext(generalBloomFilterWriter,
fileContext.getCellComparator());
- break;
-case ROWPREFIX_FIXED_LENGTH:
- bloomContext = new
RowPrefixFixedLengthBloomContext(generalBloomFilterWriter,
-fileContext.getCellComparator(), Bytes.toInt(bloomParam));
- break;
-default:
- throw new IOException(
-"Invalid Bloom filter type: " + bloomType + " (ROW or ROWCOL or
ROWPREFIX expected)");
- }
-} else {
- // Not using Bloom filters.
- this.bloomType = BloomType.NONE;
-}
+this.comparator = comparator;
+this.maxVersions = maxVersions;
+this.newVersionBehavior = newVersionBehavior;
+liveFileWriter = new SingleStoreFileWriter(fs, liveFilePath, conf,
cacheConf, bloomType,
+ maxKeys, favoredNodes, fileContext, shouldDropCacheBehind,
compactedFilesSupplier);
+ }
-// initialize delete family Bloom filter when there is NO RowCol Bloom
filter
-if (this.bloomType != BloomType.ROWCOL) {
- this.deleteFamilyBloomFilterWriter =
BloomFilterFactory.createDeleteBloomAtWrite(conf,
-cacheConf, (int) Math.min(maxKeys, Integer.MAX_VALUE), writer);
- deleteFamilyBloomContext =
-new RowBloomContext(deleteFamilyBloomFilterWriter,
fileContext.getCellComparator());
-} else {
- deleteFamilyBloomFilterWriter = null;
-}
-if (deleteFamilyBloomFilterWriter != null && LOG.isTraceEnabled()) {
- LOG.trace("Delete Family Bloom filter type for " + path + ": "
-+ deleteFamilyBloomFilterWriter.getClass().getSimpleName());
-}
+ public static boolean shouldEnableHistoricalCompactionFiles(Configuration
conf) {
+return conf.getBoolean(ENABLE_HISTORICAL_COMPACTION_FILES,
+
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545554196
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -247,7 +222,10 @@ public void appendMetadata(final long maxSequenceId, final
boolean majorCompacti
* @throws IOException problem writing to FS
*/
public void appendMobMetadata(SetMultimap mobRefSet)
throws IOException {
-writer.appendFileInfo(MOB_FILE_REFS,
MobUtils.serializeMobFileRefs(mobRefSet));
+liveFileWriter.appendMobMetadata(mobRefSet);
Review Comment:
Is there a way to prevent users enable dual file writing and mob together?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2028493399 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 47s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 31s | master passed | | +1 :green_heart: | compile | 1m 0s | master passed | | +1 :green_heart: | shadedjars | 5m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | | -0 :warning: | patch | 6m 5s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 34s | the patch passed | | +1 :green_heart: | compile | 1m 2s | the patch passed | | +1 :green_heart: | javac | 1m 2s | the patch passed | | +1 :green_heart: | shadedjars | 5m 10s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 245m 56s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 14m 55s | hbase-mapreduce in the patch passed. | | | | 285m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/26/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4b4d4ed618dd 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9adca10e9c | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/26/testReport/ | | Max. process+thread count | 5260 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/26/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2028489821 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 22s | master passed | | +1 :green_heart: | compile | 1m 5s | master passed | | +1 :green_heart: | shadedjars | 5m 56s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | | -0 :warning: | patch | 6m 48s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 56s | the patch passed | | +1 :green_heart: | compile | 1m 10s | the patch passed | | +1 :green_heart: | javac | 1m 10s | the patch passed | | +1 :green_heart: | shadedjars | 5m 48s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 223m 49s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 14m 49s | hbase-mapreduce in the patch passed. | | | | 265m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/26/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 04daf4e2b5aa 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9adca10e9c | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/26/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/26/testReport/ | | Max. process+thread count | 4470 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/26/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2028486055 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 46s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 9s | master passed | | +1 :green_heart: | compile | 1m 7s | master passed | | +1 :green_heart: | shadedjars | 5m 20s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | | -0 :warning: | patch | 6m 9s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 43s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | shadedjars | 5m 18s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 202m 59s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 12m 56s | hbase-mapreduce in the patch passed. | | | | 241m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/26/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2838688a6cf2 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9adca10e9c | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/26/testReport/ | | Max. process+thread count | 5956 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/26/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2028445194 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 35s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 9s | master passed | | +1 :green_heart: | compile | 2m 56s | master passed | | +1 :green_heart: | checkstyle | 0m 44s | master passed | | +1 :green_heart: | spotless | 0m 40s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 55s | master passed | | -0 :warning: | patch | 0m 39s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 51s | the patch passed | | +1 :green_heart: | compile | 2m 56s | the patch passed | | +1 :green_heart: | javac | 2m 56s | the patch passed | | -0 :warning: | checkstyle | 0m 34s | hbase-server: The patch generated 1 new + 39 unchanged - 0 fixed = 40 total (was 39) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 4s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 30s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 32m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/26/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 73ace762e765 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9adca10e9c | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/26/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 80 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/26/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545470519
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -256,156 +234,571 @@ public void appendMobMetadata(SetMultimap mobRefSet) throws I
public void appendTrackedTimestampsToMetadata() throws IOException {
// TODO: The StoreFileReader always converts the byte[] to TimeRange
// via TimeRangeTracker, so we should write the serialization data of
TimeRange directly.
-appendFileInfo(TIMERANGE_KEY,
TimeRangeTracker.toByteArray(timeRangeTracker));
-appendFileInfo(EARLIEST_PUT_TS, Bytes.toBytes(earliestPutTs));
+liveFileWriter.appendTrackedTimestampsToMetadata();
+if (historicalFileWriter != null) {
+ historicalFileWriter.appendTrackedTimestampsToMetadata();
+}
}
/**
* Record the earlest Put timestamp. If the timeRangeTracker is not set,
update TimeRangeTracker
* to include the timestamp of this key
*/
public void trackTimestamps(final Cell cell) {
-if (KeyValue.Type.Put.getCode() == cell.getTypeByte()) {
- earliestPutTs = Math.min(earliestPutTs, cell.getTimestamp());
+liveFileWriter.trackTimestamps(cell);
+if (historicalFileWriter != null) {
+ historicalFileWriter.trackTimestamps(cell);
}
-timeRangeTracker.includeTimestamp(cell);
}
- private void appendGeneralBloomfilter(final Cell cell) throws IOException {
-if (this.generalBloomFilterWriter != null) {
- /*
- *
http://2.bp.blogspot.com/_Cib_A77V54U/StZMrzaKufI/ADo/ZhK7bGoJdMQ/s400/KeyValue.png
- * Key = RowLen + Row + FamilyLen + Column [Family + Qualifier] +
Timestamp 3 Types of
- * Filtering: 1. Row = Row 2. RowCol = Row + Qualifier 3.
RowPrefixFixedLength = Fixed Length
- * Row Prefix
- */
- bloomContext.writeBloom(cell);
+ @Override
+ public void beforeShipped() throws IOException {
+liveFileWriter.beforeShipped();
+if (historicalFileWriter != null) {
+ historicalFileWriter.beforeShipped();
}
}
- private void appendDeleteFamilyBloomFilter(final Cell cell) throws
IOException {
-if (!PrivateCellUtil.isDeleteFamily(cell) &&
!PrivateCellUtil.isDeleteFamilyVersion(cell)) {
- return;
-}
+ public Path getPath() {
+return liveFileWriter.getPath();
Review Comment:
I had to introduce another method called getPaths to get the list of paths,
that is , up to two paths, one for the live file and the other for the
historical file. getPath is currently called by the compactions that do not
support dual file writing yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545469228
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -256,156 +234,571 @@ public void appendMobMetadata(SetMultimap mobRefSet) throws I
public void appendTrackedTimestampsToMetadata() throws IOException {
// TODO: The StoreFileReader always converts the byte[] to TimeRange
// via TimeRangeTracker, so we should write the serialization data of
TimeRange directly.
-appendFileInfo(TIMERANGE_KEY,
TimeRangeTracker.toByteArray(timeRangeTracker));
-appendFileInfo(EARLIEST_PUT_TS, Bytes.toBytes(earliestPutTs));
+liveFileWriter.appendTrackedTimestampsToMetadata();
Review Comment:
I did not change the existing behavior. I refactored the code such that the
exiting StoreFileWriter implementation was moved to the new nested class called
SingleStoreFileWriter. StoreFileWriter has now two SingleStoreFileWriter
objects, liveFileWriter, and historicalFileWriter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545467925
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -247,7 +222,10 @@ public void appendMetadata(final long maxSequenceId, final
boolean majorCompacti
* @throws IOException problem writing to FS
*/
public void appendMobMetadata(SetMultimap mobRefSet)
throws IOException {
-writer.appendFileInfo(MOB_FILE_REFS,
MobUtils.serializeMobFileRefs(mobRefSet));
+liveFileWriter.appendMobMetadata(mobRefSet);
Review Comment:
By default all files are live file when the dual file writing is not
enabled. Dual file writing is not supported for mob in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545467547
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -235,10 +210,10 @@ private byte[]
toCompactionEventTrackerBytes(Collection storeFiles)
*/
public void appendMetadata(final long maxSequenceId, final boolean
majorCompaction,
final long mobCellsCount) throws IOException {
-writer.appendFileInfo(MAX_SEQ_ID_KEY, Bytes.toBytes(maxSequenceId));
-writer.appendFileInfo(MAJOR_COMPACTION_KEY,
Bytes.toBytes(majorCompaction));
-writer.appendFileInfo(MOB_CELLS_COUNT, Bytes.toBytes(mobCellsCount));
-appendTrackedTimestampsToMetadata();
+liveFileWriter.appendMetadata(maxSequenceId, majorCompaction,
mobCellsCount);
Review Comment:
I did not change the behavior. This change is due to refactoring the code.
We do not support dual file writing for mob yet as mentioned in the previous
comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545467119
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -105,72 +132,44 @@ public class StoreFileWriter implements CellSink,
ShipperListener {
* @param fileContextThe HFile context
* @param shouldDropCacheBehind Drop pages written to page cache after
writing the store file.
* @param compactedFilesSupplier Returns the {@link HStore} compacted files
which not archived
+ * @param comparator Cell comparator
+ * @param maxVersionsmax cell versions
+ * @param newVersionBehavior enable new version behavior
* @throws IOException problem writing to FS
*/
- private StoreFileWriter(FileSystem fs, Path path, final Configuration conf,
CacheConfig cacheConf,
-BloomType bloomType, long maxKeys, InetSocketAddress[] favoredNodes,
HFileContext fileContext,
-boolean shouldDropCacheBehind, Supplier>
compactedFilesSupplier)
-throws IOException {
+ private StoreFileWriter(FileSystem fs, Path liveFilePath, Path
historicalFilePath,
+final Configuration conf, CacheConfig cacheConf, BloomType bloomType, long
maxKeys,
+InetSocketAddress[] favoredNodes, HFileContext fileContext, boolean
shouldDropCacheBehind,
+Supplier> compactedFilesSupplier, CellComparator
comparator,
+int maxVersions, boolean newVersionBehavior) throws IOException {
+this.fs = fs;
+this.historicalFilePath = historicalFilePath;
+this.conf = conf;
+this.cacheConf = cacheConf;
+this.bloomType = bloomType;
+this.maxKeys = maxKeys;
+this.favoredNodes = favoredNodes;
+this.fileContext = fileContext;
+this.shouldDropCacheBehind = shouldDropCacheBehind;
this.compactedFilesSupplier = compactedFilesSupplier;
-this.timeRangeTracker =
TimeRangeTracker.create(TimeRangeTracker.Type.NON_SYNC);
-// TODO : Change all writers to be specifically created for compaction
context
-writer =
- HFile.getWriterFactory(conf, cacheConf).withPath(fs,
path).withFavoredNodes(favoredNodes)
-
.withFileContext(fileContext).withShouldDropCacheBehind(shouldDropCacheBehind).create();
-
-generalBloomFilterWriter =
BloomFilterFactory.createGeneralBloomAtWrite(conf, cacheConf,
- bloomType, (int) Math.min(maxKeys, Integer.MAX_VALUE), writer);
-
-if (generalBloomFilterWriter != null) {
- this.bloomType = bloomType;
- this.bloomParam = BloomFilterUtil.getBloomFilterParam(bloomType, conf);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Bloom filter type for " + path + ": " + this.bloomType + ",
param: "
- + (bloomType == BloomType.ROWPREFIX_FIXED_LENGTH
-? Bytes.toInt(bloomParam)
-: Bytes.toStringBinary(bloomParam))
- + ", " + generalBloomFilterWriter.getClass().getSimpleName());
- }
- // init bloom context
- switch (bloomType) {
-case ROW:
- bloomContext =
-new RowBloomContext(generalBloomFilterWriter,
fileContext.getCellComparator());
- break;
-case ROWCOL:
- bloomContext =
-new RowColBloomContext(generalBloomFilterWriter,
fileContext.getCellComparator());
- break;
-case ROWPREFIX_FIXED_LENGTH:
- bloomContext = new
RowPrefixFixedLengthBloomContext(generalBloomFilterWriter,
-fileContext.getCellComparator(), Bytes.toInt(bloomParam));
- break;
-default:
- throw new IOException(
-"Invalid Bloom filter type: " + bloomType + " (ROW or ROWCOL or
ROWPREFIX expected)");
- }
-} else {
- // Not using Bloom filters.
- this.bloomType = BloomType.NONE;
-}
+this.comparator = comparator;
+this.maxVersions = maxVersions;
+this.newVersionBehavior = newVersionBehavior;
+liveFileWriter = new SingleStoreFileWriter(fs, liveFilePath, conf,
cacheConf, bloomType,
+ maxKeys, favoredNodes, fileContext, shouldDropCacheBehind,
compactedFilesSupplier);
+ }
-// initialize delete family Bloom filter when there is NO RowCol Bloom
filter
-if (this.bloomType != BloomType.ROWCOL) {
- this.deleteFamilyBloomFilterWriter =
BloomFilterFactory.createDeleteBloomAtWrite(conf,
-cacheConf, (int) Math.min(maxKeys, Integer.MAX_VALUE), writer);
- deleteFamilyBloomContext =
-new RowBloomContext(deleteFamilyBloomFilterWriter,
fileContext.getCellComparator());
-} else {
- deleteFamilyBloomFilterWriter = null;
-}
-if (deleteFamilyBloomFilterWriter != null && LOG.isTraceEnabled()) {
- LOG.trace("Delete Family Bloom filter type for " + path + ": "
-+ deleteFamilyBloomFilterWriter.getClass().getSimpleName());
-}
+ public static boolean shouldEnableHistoricalCompactionFiles(Configuration
conf) {
+return conf.getBoolean(ENABLE_HISTORICAL_COMPACTION_FILES,
+
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545464762
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStoreFile.java:
##
@@ -138,6 +140,12 @@ public class HStoreFile implements StoreFile {
// Indicates if the file got compacted
private volatile boolean compactedAway = false;
+ // Indicate if the file contains historical cell versions. This is used when
+ // hbase.enable.historical.compaction.files is set to true. In that case,
compactions
+ // can generate two files, one with the live cell versions and the other
with the remaining
+ // (historical) cell versions.
+ private volatile boolean isHistorical = false;
Review Comment:
It looks like I need to add more comments for this. If isHistorical is true
then the hfile is historical. Historical files are skipped for regular (not
raw) scans scanning latest row versions. When
hbase.enable.historical.compaction.files is false, the historical flag will be
false for all files. This means all files will be treated as live files, and
thus there is no need to track them. Historical files are generated only when
hbase.enable.historical.compaction.files is true. Only when
hbase.enable.historical.compaction.files is true, we enable live file file
tracking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545462036
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java:
##
@@ -86,13 +111,20 @@ public Collection getCompactedfiles() {
}
@Override
- public void insertNewFiles(Collection sfs) {
+ public void insertNewFiles(Collection sfs) throws IOException {
+if (enableLiveFileTracking) {
+ this.liveStoreFiles =
ImmutableList.sortedCopyOf(getStoreFileComparator(),
Review Comment:
DefaultStoreFileManager is not thread safe, the caller is responsible for
locking. In this case, StoreEngine acquires a write lock on the store before
calling the methods of StoreFileManager. This prevents the consistency issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache9 commented on code in PR #5545:
URL: https://github.com/apache/hbase/pull/5545#discussion_r1545362732
##
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java:
##
@@ -105,72 +132,44 @@ public class StoreFileWriter implements CellSink,
ShipperListener {
* @param fileContextThe HFile context
* @param shouldDropCacheBehind Drop pages written to page cache after
writing the store file.
* @param compactedFilesSupplier Returns the {@link HStore} compacted files
which not archived
+ * @param comparator Cell comparator
+ * @param maxVersionsmax cell versions
+ * @param newVersionBehavior enable new version behavior
* @throws IOException problem writing to FS
*/
- private StoreFileWriter(FileSystem fs, Path path, final Configuration conf,
CacheConfig cacheConf,
-BloomType bloomType, long maxKeys, InetSocketAddress[] favoredNodes,
HFileContext fileContext,
-boolean shouldDropCacheBehind, Supplier>
compactedFilesSupplier)
-throws IOException {
+ private StoreFileWriter(FileSystem fs, Path liveFilePath, Path
historicalFilePath,
+final Configuration conf, CacheConfig cacheConf, BloomType bloomType, long
maxKeys,
+InetSocketAddress[] favoredNodes, HFileContext fileContext, boolean
shouldDropCacheBehind,
+Supplier> compactedFilesSupplier, CellComparator
comparator,
+int maxVersions, boolean newVersionBehavior) throws IOException {
+this.fs = fs;
+this.historicalFilePath = historicalFilePath;
+this.conf = conf;
+this.cacheConf = cacheConf;
+this.bloomType = bloomType;
+this.maxKeys = maxKeys;
+this.favoredNodes = favoredNodes;
+this.fileContext = fileContext;
+this.shouldDropCacheBehind = shouldDropCacheBehind;
this.compactedFilesSupplier = compactedFilesSupplier;
-this.timeRangeTracker =
TimeRangeTracker.create(TimeRangeTracker.Type.NON_SYNC);
-// TODO : Change all writers to be specifically created for compaction
context
-writer =
- HFile.getWriterFactory(conf, cacheConf).withPath(fs,
path).withFavoredNodes(favoredNodes)
-
.withFileContext(fileContext).withShouldDropCacheBehind(shouldDropCacheBehind).create();
-
-generalBloomFilterWriter =
BloomFilterFactory.createGeneralBloomAtWrite(conf, cacheConf,
- bloomType, (int) Math.min(maxKeys, Integer.MAX_VALUE), writer);
-
-if (generalBloomFilterWriter != null) {
- this.bloomType = bloomType;
- this.bloomParam = BloomFilterUtil.getBloomFilterParam(bloomType, conf);
- if (LOG.isTraceEnabled()) {
-LOG.trace("Bloom filter type for " + path + ": " + this.bloomType + ",
param: "
- + (bloomType == BloomType.ROWPREFIX_FIXED_LENGTH
-? Bytes.toInt(bloomParam)
-: Bytes.toStringBinary(bloomParam))
- + ", " + generalBloomFilterWriter.getClass().getSimpleName());
- }
- // init bloom context
- switch (bloomType) {
-case ROW:
- bloomContext =
-new RowBloomContext(generalBloomFilterWriter,
fileContext.getCellComparator());
- break;
-case ROWCOL:
- bloomContext =
-new RowColBloomContext(generalBloomFilterWriter,
fileContext.getCellComparator());
- break;
-case ROWPREFIX_FIXED_LENGTH:
- bloomContext = new
RowPrefixFixedLengthBloomContext(generalBloomFilterWriter,
-fileContext.getCellComparator(), Bytes.toInt(bloomParam));
- break;
-default:
- throw new IOException(
-"Invalid Bloom filter type: " + bloomType + " (ROW or ROWCOL or
ROWPREFIX expected)");
- }
-} else {
- // Not using Bloom filters.
- this.bloomType = BloomType.NONE;
-}
+this.comparator = comparator;
+this.maxVersions = maxVersions;
+this.newVersionBehavior = newVersionBehavior;
+liveFileWriter = new SingleStoreFileWriter(fs, liveFilePath, conf,
cacheConf, bloomType,
+ maxKeys, favoredNodes, fileContext, shouldDropCacheBehind,
compactedFilesSupplier);
+ }
-// initialize delete family Bloom filter when there is NO RowCol Bloom
filter
-if (this.bloomType != BloomType.ROWCOL) {
- this.deleteFamilyBloomFilterWriter =
BloomFilterFactory.createDeleteBloomAtWrite(conf,
-cacheConf, (int) Math.min(maxKeys, Integer.MAX_VALUE), writer);
- deleteFamilyBloomContext =
-new RowBloomContext(deleteFamilyBloomFilterWriter,
fileContext.getCellComparator());
-} else {
- deleteFamilyBloomFilterWriter = null;
-}
-if (deleteFamilyBloomFilterWriter != null && LOG.isTraceEnabled()) {
- LOG.trace("Delete Family Bloom filter type for " + path + ": "
-+ deleteFamilyBloomFilterWriter.getClass().getSimpleName());
-}
+ public static boolean shouldEnableHistoricalCompactionFiles(Configuration
conf) {
+return conf.getBoolean(ENABLE_HISTORICAL_COMPACTION_FILES,
+
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2001897341 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 30s | master passed | | +1 :green_heart: | compile | 1m 1s | master passed | | +1 :green_heart: | shadedjars | 5m 10s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | | -0 :warning: | patch | 6m 7s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 31s | the patch passed | | +1 :green_heart: | compile | 1m 0s | the patch passed | | +1 :green_heart: | javac | 1m 0s | the patch passed | | +1 :green_heart: | shadedjars | 5m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 232m 38s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 15m 2s | hbase-mapreduce in the patch passed. | | | | 272m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2c78ec9342d4 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9361ae506a | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/testReport/ | | Max. process+thread count | 5695 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2001894215 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 7s | master passed | | +1 :green_heart: | compile | 1m 5s | master passed | | +1 :green_heart: | shadedjars | 5m 48s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | | -0 :warning: | patch | 6m 39s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 58s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | shadedjars | 5m 50s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 218m 57s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 14m 19s | hbase-mapreduce in the patch passed. | | | | 259m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5e8e264d8a26 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9361ae506a | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/testReport/ | | Max. process+thread count | 5229 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2001888007 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 45s | master passed | | +1 :green_heart: | compile | 1m 8s | master passed | | +1 :green_heart: | shadedjars | 5m 19s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | | -0 :warning: | patch | 6m 9s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 43s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | shadedjars | 5m 21s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 201m 1s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 12m 56s | hbase-mapreduce in the patch passed. | | | | 238m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 4ba6ecea3fe6 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9361ae506a | | Default Java | Eclipse Adoptium-17.0.10+7 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/testReport/ | | Max. process+thread count | 5703 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2001502697 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 46s | master passed | | +1 :green_heart: | compile | 2m 54s | master passed | | +1 :green_heart: | checkstyle | 0m 44s | master passed | | +1 :green_heart: | spotless | 0m 39s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 51s | master passed | | -0 :warning: | patch | 0m 39s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | compile | 2m 54s | the patch passed | | +1 :green_heart: | javac | 2m 54s | the patch passed | | -0 :warning: | checkstyle | 0m 33s | hbase-server: The patch generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 3s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 30m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux ff5eb6c97e4b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9361ae506a | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 80 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/25/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2001352366 Added tests for new version behavior and found a bug in HBase and create the [jira](https://issues.apache.org/jira/browse/HBASE-28442) for that. One of the tests fails because of this bug. The PR is now ready for your review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2000699516 > @kadirozde i just verified with HBase 2.6 (branch-2) that the flush writes only maxVersions versions to the new HFile. Hence, if the max version is 1, and if we write 2 versions of the cell, only the latest cell is written to the HFile. > So we should be good for the first case. > > At this point, we can add some tests for newVersionBehavior, and we should be good for the above concern. @virajjasani, Thank you for verifying this. I had the wrong assumption about the memstore behavior and thus thought that there were issues in hbase with handling delete version markers. I will add new tests and wrap up this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
virajjasani commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2000651680 > i just verified with HBase 2.6 (branch-2) that the flush writes only maxVersions versions to the new HFile. Btw this also means that raw scan (that reads all versions) can have different results while scanning the cell with multiple versions because if we do raw scan before flush, we will get all versions written by user (regardless of max versions set at the table descriptor), however if we do raw scan after flush, we will get only max versions num of versions for the given cell. It is expected or perhaps we should document that anyone doing raw scan should read only max versions num of versions during the scan, otherwise (if they read all versions during raw scan) they can observe different results before and after flush operation. This behavior has nothing to do with this PR though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
virajjasani commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2000634670 > @Apache9, @virajjasani, @bbeaudreault , This PR is not done yet. I just realized that I need to add testing for newVersionBehavior. > > Also, I need to discuss the following case: > > Assume that for a given cell, two versions inserted and max versions is set to 1. If memory compaction is not enabled then I expect that both versions will be written to a new hfile (hf1) during flush even though the second version is redundant (is that true? I need to verify this). Now during minor compaction, the latest version will be written to a new live file (hf2) and the redundant version to a new historical file (hf3). Assume that a delete version marker is inserted for the latest version. This delete marker will be written to a new hfile (hf4). This delete marker will mask mask the latest version, and regular scans for the latest versions will not return any of the versions of this cell as latest version is masked by the delete marker and the redundant version is in the historical file (hf3) will be omitted by these scans. When the major compaction happens I expect that the redundant version should be revived and will be written to a new live file (hf5). Now the redunda nt version would be visible to regular scans. Please let me know if any of these is incorrect. > > Please note this should not happen with newVersionBehavior as the deleted versions are considered toward total version count. Should we enable this feature only when newVersionBehavior is enabled? @kadirozde i just verified with HBase 2.6 (branch-2) that the flush writes only maxVersions versions to the new HFile. Hence, if the max version is 1, and if we write 2 versions of the cell, only the latest cell is written to the HFile. So we should be good for the first case. At this point, we can add some tests for newVersionBehavior, and we should be good for the above concern. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2000602763 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 30s | master passed | | +1 :green_heart: | compile | 1m 2s | master passed | | +1 :green_heart: | shadedjars | 5m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | | -0 :warning: | patch | 6m 5s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 31s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 5m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 247m 49s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 14m 52s | hbase-mapreduce in the patch passed. | | | | 287m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/24/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a36f7c68bc4f 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9361ae506a | | Default Java | Temurin-1.8.0_352-b08 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/24/testReport/ | | Max. process+thread count | 6033 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/24/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2000584582 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 58s | master passed | | +1 :green_heart: | compile | 1m 5s | master passed | | +1 :green_heart: | shadedjars | 5m 52s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | | -0 :warning: | patch | 6m 43s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 54s | the patch passed | | +1 :green_heart: | compile | 1m 9s | the patch passed | | +1 :green_heart: | javac | 1m 9s | the patch passed | | +1 :green_heart: | shadedjars | 5m 54s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 221m 16s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 14m 32s | hbase-mapreduce in the patch passed. | | | | 262m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/24/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c8b0ec027e2b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9361ae506a | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/24/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/24/testReport/ | | Max. process+thread count | 5261 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/24/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
virajjasani commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2000579921 > Assume that for a given cell, two versions inserted and max versions is set to 1. If memory compaction is not enabled then I expect that both versions will be written to a new hfile (hf1) during flush even though the second version is redundant (is that true? I need to verify this). If table descriptor level max version is set to 1, during flush only latest version will be written to the HFile hf1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2000576912 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 10s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 42s | master passed | | +1 :green_heart: | compile | 1m 9s | master passed | | +1 :green_heart: | shadedjars | 5m 23s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | | -0 :warning: | patch | 6m 13s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 45s | the patch passed | | +1 :green_heart: | compile | 1m 8s | the patch passed | | +1 :green_heart: | javac | 1m 8s | the patch passed | | +1 :green_heart: | shadedjars | 5m 20s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 214m 52s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 13m 18s | hbase-mapreduce in the patch passed. | | | | 252m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/24/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0a0e17d1ab46 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9361ae506a | | Default Java | Eclipse Adoptium-17.0.10+7 | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/24/testReport/ | | Max. process+thread count | 5202 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/24/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2000437590 @Apache9, @virajjasani, @bbeaudreault , This PR is not done yet. I just realized that I need to add testing for newVersionBehavior. Also, I need to discuss the following case: Assume that for a given cell, two versions inserted and max versions is set to 1. If memory compaction is not enabled then I expect that both versions will be written to a new hfile (hf1) during flush even though the second version is redundant (is that true? I need to verify this). Now during minor compaction, the latest version will be written to a new live file (hf2) and the redundant version to a new historical file (hf3). Assume that a delete version marker is inserted for the latest version. This delete marker will be written to a new hfile (hf4). This delete marker will mask mask the latest version, and regular scans for the latest versions will not return any of the versions of this cell as latest version is masked by the delete marker and the redundant version is in the historical file (hf3) will be omitted by these scans. When the major compaction happens I expect that the redundant version should be revived and will be written to a new live file (hf5). Now the redundant version would be visible to regular scans. Please let me know if any of this incorrect. Please note this should not happen with newVersionBehavior as the deleted versions are considered toward total version count. Should we enable this feature only when newVersionBehavior is enabled? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
Apache-HBase commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2000243609 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 42s | master passed | | +1 :green_heart: | compile | 3m 17s | master passed | | +1 :green_heart: | checkstyle | 0m 51s | master passed | | +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 1m 58s | master passed | | -0 :warning: | patch | 0m 46s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 1s | the patch passed | | +1 :green_heart: | compile | 2m 54s | the patch passed | | +1 :green_heart: | javac | 2m 54s | the patch passed | | -0 :warning: | checkstyle | 0m 34s | hbase-server: The patch generated 1 new + 37 unchanged - 0 fixed = 38 total (was 37) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 3s | Patch does not cause any errors with Hadoop 3.3.6. | | +1 :green_heart: | spotless | 0m 40s | patch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. | | | | 33m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/24/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5545 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux 44e74913fe23 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 9361ae506a | | Default Java | Eclipse Adoptium-11.0.17+8 | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/24/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-server hbase-mapreduce U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5545/24/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-2000166912 @bbeaudreault, @virajjasani, I updated the PR such that the historical files will be generated only with default store engine and default compactor. In other cases, historical files will not be generated and hbase.enable.historical.compaction.files will be ignored silently. I will file jiras to support historical files for other types of compactions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
virajjasani commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-213796 I can also re-review the PR since my last review is now stale. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-25972 Dual File Compaction [hbase]
kadirozde commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-209812 > @Apache9 @kadirozde How close is this to ready? I think I'm reading to cut the first RC0 of 2.6.0 on Monday, but this might be a nice addition to the release. We've delayed 2.6.0 for a long time so I'd rather not delay it much longer though. In the current PR, this feature is enabled for all compaction types but testing is done only using default compaction. Since the change is at the store file writer level, we think it should work for all. I can add checks to disable it for other compactions as @Apache9 suggested earlier, for the interest of the time. I can get this done today. However, after the store file writer restructuring done yesterday, the PR has not been reviewed yet. So, completing the review may some time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
