szetszwo commented on code in PR #9689:
URL: https://github.com/apache/ozone/pull/9689#discussion_r2755599398
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHAManagerImpl.java:
##########
@@ -95,10 +95,31 @@ public SCMHAManagerImpl(final ConfigurationSource conf,
}
@VisibleForTesting
- protected SCMSnapshotProvider newScmSnapshotProvider(StorageContainerManager
storageContainerManager) {
- return new SCMSnapshotProvider(storageContainerManager.getConfiguration(),
+ protected SCMSnapshotProvider newScmSnapshotProvider(
+ StorageContainerManager storageContainerManager) {
Review Comment:
This method is not used in test. Let's remove @ VisibleForTesting
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMSnapshotProvider.java:
##########
@@ -55,24 +55,111 @@ public class SCMSnapshotProvider {
private final CertificateClient scmCertificateClient;
+ /**
+ * Startup options for SCM snapshot provider.
+ */
+ public enum StartupOption {
+ /**
+ * FORMAT mode: Ratis snapshot directory should not exist.
+ * Will create and initialize a new Ratis snapshot directory.
+ */
+ FORMAT,
+ /**
+ * NORMAL mode: Ratis snapshot directory should already exist.
+ * Will read from existing Ratis snapshot directory.
+ */
+ NORMAL
+ }
+
+ /**
+ * Creates SCMSnapshotProvider with default NORMAL startup option.
+ * This constructor is used in most scenarios where the Ratis snapshot
+ * directory is expected to already exist.
+ *
+ * @param conf Configuration source
+ * @param peerNodes List of peer SCM nodes
+ * @param scmCertificateClient Certificate client for secure communication
+ */
public SCMSnapshotProvider(ConfigurationSource conf,
List<SCMNodeDetails> peerNodes,
CertificateClient scmCertificateClient) {
Review Comment:
This method is used only in TestSCMSnapshotProvider, let's remove it and
change the test.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHAManagerImpl.java:
##########
@@ -95,10 +95,31 @@ public SCMHAManagerImpl(final ConfigurationSource conf,
}
@VisibleForTesting
- protected SCMSnapshotProvider newScmSnapshotProvider(StorageContainerManager
storageContainerManager) {
- return new SCMSnapshotProvider(storageContainerManager.getConfiguration(),
+ protected SCMSnapshotProvider newScmSnapshotProvider(
+ StorageContainerManager storageContainerManager) {
+
+ // Determine startup option based on snapshot directory existence
+ String snapshotDir = SCMHAUtils.getSCMRatisSnapshotDirectory(
+ storageContainerManager.getConfiguration());
+ File snapshotDirFile = new File(snapshotDir);
+
+ SCMSnapshotProvider.StartupOption startupOption;
+ if (snapshotDirFile.exists()) {
Review Comment:
Let's use
```java
if (Files.isDirectory(Paths.get(snapshotDir))) {
```
##########
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-repair-tools.sh:
##########
@@ -98,5 +98,14 @@ echo "Test keys created"
echo "Restarting OM after key creation to flush and generate sst files"
docker restart "${om_container}"
+wait_for_om_leader
+
+echo "Deleting test keys to create tombstones"
+docker exec "${om_container}" bash -c "kinit -kt
/etc/security/keytabs/testuser.keytab testuser/[email protected] && ozone fs -rm
-R -skipTrash ofs://omservice/vol1/bucket1"
+echo "Test keys deleted"
+
+echo "Restarting OM after key deletion to flush tombstones to sst files"
+docker restart "${om_container}"
+wait_for_om_leader
Review Comment:
This change seems unrelated? If yes, please revert it.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java:
##########
@@ -1340,6 +1350,13 @@ private static SCMStorageConfig
initializeRatis(OzoneConfiguration conf)
final SCMHANodeDetails haDetails = SCMHANodeDetails.loadSCMHAConfig(conf,
storageConfig);
SCMRatisServerImpl.initialize(storageConfig.getClusterID(),
storageConfig.getScmId(), haDetails.getLocalNodeDetails(), conf);
+
+ // Create Ratis snapshot directory during initialization
+ // This ensures the directory exists before SCM starts normally
+ String snapshotDir = SCMHAUtils.getSCMRatisSnapshotDirectory(conf);
Review Comment:
snapshotDir should be created only for scmInit or scmBootstrap. So, it
should not be created here.
##########
hadoop-ozone/dist/src/main/smoketest/repair/om-compact.robot:
##########
@@ -36,19 +36,20 @@ Compact OM DB Column Family
[Arguments] ${column_family}
Execute ozone repair om compact --cf=${column_family} --service-id
${OM_SERVICE_ID} --node-id om1
+SST Size Should Be Reduced
+ [Arguments] ${original_size}
+ ${current_size} = Get OM DB SST Files Size
+ Should Be True ${current_size} < ${original_size}
+ ... OM DB size should be reduced after compaction. Before:
${original_size}, After: ${current_size}
+
*** Test Cases ***
Testing OM DB Size Reduction After Compaction
- # Test keys are already created and flushed
- # Delete keys to create tombstones that need compaction
- Delete Test Keys
-
+ # Test keys are already created, deleted and flushed by the test script
${size_before_compaction} = Get OM DB SST Files Size
Compact OM DB Column Family fileTable
Compact OM DB Column Family deletedTable
Compact OM DB Column Family deletedDirectoryTable
-
- ${size_after_compaction} = Get OM DB SST Files Size
- Should Be True ${size_after_compaction} < ${size_before_compaction}
- ... OM DB size should be reduced after compaction. Before:
${size_before_compaction}, After: ${size_after_compaction}
+ # Compaction is asynchronous, poll until size is reduced
+ Wait Until Keyword Succeeds 60sec 5sec SST Size Should Be Reduced
${size_before_compaction}
Review Comment:
This change seems unrelated? If yes, please revert it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
