This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3cd2aa4a6bb [fix](profile) Harden the profile completion state against
lost reports and concurrent rendering (#69044)
3cd2aa4a6bb is described below
commit 3cd2aa4a6bbe458f76594b98cb978a821446653d
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Mon Oct 19 07:34:42 2026 +0100
[fix](profile) Harden the profile completion state against lost reports and
concurrent rendering (#69044)
### What problem does this PR solve?
Issue Number: None
Related PR: #66227, #64392
Problem Summary:
**In short.** A query profile shows a completion state (RUNNING,
COLLECTING, COMPLETE, INCOMPLETE), and regression tests wait for
COMPLETE before they read profile counters. On master the state can be
wrong in three ways: a profile whose BE reports were dropped after a
failed spill says COMPLETE, a profile rendered while it is being spilled
can keep COLLECTING for good, and a profile can say COMPLETE before the
last BE report is visible in its per-pipeline section. This PR ports the
fixes for all three from the profile part of branch-4.1 #66227.
**Background.** #64392 added the completion state on master.
`ExecutionProfile` counts the BEs that sent a final report for each
fragment, and `Profile` derives the state from that: RUNNING until the
query finishes, then COLLECTING until every BE has sent its final
report, then COMPLETE. When `ProfileManager` spills a profile to disk it
writes a terminal state instead (COMPLETE, or INCOMPLETE when reports
are still missing), and after the spill it releases the in-memory
execution profiles. The state is printed in the profile summary
(`Profile Completion State: ...`) and in the query profile list.
branch-4.1 got the same mechanism inside #66227, together with the three
fixes below; master never got the fixes.
**The problem, and what it cost.**
| Case | master | with this PR |
|---|---|---|
| The spill fails (for example the profile directory is not writable).
`writeProfileToStorage` still releases the reports, and the next spill
writes the profile again. | The state is computed over an empty report
list, so it shows COMPLETE, and the next spill stores COMPLETE for a
profile whose BE reports are gone. | INCOMPLETE, in memory and in the
stored profile. |
| The profile is rendered (`SHOW QUERY PROFILE`, the web UI) while it is
being spilled. Rendering writes the display state into the same summary
field the spill has just set to the terminal state. | Rendered before
serialization: the stored file says COLLECTING. Rendered after
serialization but before the storage path is set: the file says
INCOMPLETE, but the profile in memory says COLLECTING from then on.
Either way a poller waiting for a terminal state never gets one. | The
terminal state is frozen before serialization and returned from then on;
a failed write drops the frozen state so a retry recomputes it. |
| A BE sends its final report. `updateProfile` marks the BE done under
the lock and attaches its pipeline nodes after releasing the lock. | A
reader that sees COMPLETE in between can render, or spill, a profile
whose per-pipeline section lacks that report while the merged section
already has it. | The nodes are attached under the same lock, before the
BE is marked done. |
**How this PR fixes it.**
- `Profile.releaseMemory()` records that the reports were released; from
then on both the display state and the storage state are INCOMPLETE.
- `Profile.writeToStorage()` freezes the storage state before it
serializes the summary, and `getProfileCompletionState()` returns the
frozen state first. The two state updates share a dedicated lock, and a
failed write clears the frozen state.
- `ExecutionProfile.updateMultiBeProfile()` attaches the report's
pipeline nodes to the fragment profile inside the write lock, before it
adds the BE to the done set.
The code is the same as on branch-4.1 and branch-4.2, except for that
lock: branch-4.x makes the two updates `synchronized` on the `Profile`
monitor, which deadlocks. `ProfileManager` renders a profile under its
read lock and rendering then waits for the monitor, while
`Profile.updateSummary()` holds the monitor and waits for the manager's
write lock in `pushProfile()`.
**Results.** `ProfileCompletionStateTest` (new) and
`ProfileManagerTest#testFailedSpillRemainsIncompleteAfterMemoryRelease`
(new) cover the failed spill, a failed write followed by a successful
one, rendering paused before and after serialization, the state after
reports are released, multiple pipeline nodes from one BE, and a stale
non-final report after the final one.
`renderingDoesNotWaitForProfileMonitor` renders a profile while the test
holds the `Profile` monitor; with the monitor-based updates of
branch-4.x it times out. With the fix, all 36 tests in
`ProfileCompletionStateTest` and `ProfileManagerTest` pass. With
master's `Profile` and `ExecutionProfile`,
`testFailedSpillRemainsIncompleteAfterMemoryRelease` gets COMPLETE and
`renderingDuringStoragePreservesTerminalState` gets COLLECTING where
both expect INCOMPLETE; removing only the reset after a failed write
makes `failedStorageDoesNotFreezeCompletionState` get INCOMPLETE where
it expects COLLECTING. The order in which `ExecutionProfile` publishes
the nodes is a race and has no deterministic test.
Co-authored-by: Fangyuan Deng <[email protected]>
---
.../doris/common/profile/ExecutionProfile.java | 11 +-
.../org/apache/doris/common/profile/Profile.java | 34 ++-
.../common/profile/ProfileCompletionStateTest.java | 244 +++++++++++++++++++++
.../doris/common/profile/ProfileManagerTest.java | 29 +++
4 files changed, 311 insertions(+), 7 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecutionProfile.java
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecutionProfile.java
index 067c695e1d8..48834936821 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecutionProfile.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/ExecutionProfile.java
@@ -168,13 +168,17 @@ public class ExecutionProfile {
}
private boolean updateMultiBeProfile(int fragmentId, TNetworkAddress
backendHBAddress,
- List<RuntimeProfile> taskProfile, boolean isDone) {
+ List<RuntimeProfile> taskProfile, List<RuntimeProfile>
profileNodes, boolean isDone) {
multiBeProfileLock.writeLock().lock();
try {
if (!isDone &&
fragmentIdDoneBackends.get(fragmentId).contains(backendHBAddress)) {
return false;
}
multiBeProfile.get(fragmentId).put(backendHBAddress, taskProfile);
+ // Publish both rendered forms before completion becomes visible
to profile readers.
+ for (RuntimeProfile profileNode : profileNodes) {
+ fragmentProfiles.get(fragmentId).addChild(profileNode, true);
+ }
if (isDone) {
fragmentIdDoneBackends.get(fragmentId).add(backendHBAddress);
}
@@ -303,12 +307,9 @@ public class ExecutionProfile {
profileNode.setIsDone(isDone);
profileNodes.add(profileNode);
}
- if (!updateMultiBeProfile(fragmentId, backendHBAddress,
taskProfile, isDone)) {
+ if (!updateMultiBeProfile(fragmentId, backendHBAddress,
taskProfile, profileNodes, isDone)) {
continue;
}
- for (RuntimeProfile profileNode : profileNodes) {
- fragmentProfiles.get(fragmentId).addChild(profileNode, true);
- }
}
LOG.info("Profile update finished query: {} fragments: {} isDone: {}",
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
index b6da0447c31..efc695e3518 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/profile/Profile.java
@@ -117,6 +117,12 @@ public class Profile {
// 1. profile is stored to storage
// 2. or profile is loaded from storage
private String profileStoragePath = "";
+ private volatile String storageProfileCompletionState;
+ private volatile boolean executionProfilesReleased;
+ // Serializes the storage and display updates of the completion state. Not
the Profile monitor:
+ // ProfileManager renders under its read lock, while updateSummary() holds
the monitor and then
+ // takes the manager's write lock.
+ private final Object completionStateLock = new Object();
// isQueryFinished means the coordinator or stmt executor is finished.
// does not mean the profile report has finished, since the report is
async.
// finish of collection of profile is marked by isCompleted of
ExecutionProfiles.
@@ -565,6 +571,9 @@ public class Profile {
}
public void releaseMemory() {
+ // The manager also releases reports after a failed spill. An empty
list then means
+ // lost reports, not successful collection; retain that distinction
across retries.
+ executionProfilesReleased = true;
this.executionProfiles.clear();
this.changedSessionVarCache = "";
this.physicalPlan = null;
@@ -627,7 +636,13 @@ public class Profile {
return !Strings.isNullOrEmpty(profileStoragePath);
}
+ // Query completion precedes asynchronous BE reports; persisted profiles
must retain
+ // their terminal state even after the in-memory reports have been
released.
public String getProfileCompletionState() {
+ String storageState = storageProfileCompletionState;
+ if (storageState != null) {
+ return storageState;
+ }
if (profileHasBeenStored()) {
String storedState =
summaryProfile.getSummary().getInfoString(SummaryProfile.PROFILE_COMPLETION_STATE);
if (!Strings.isNullOrEmpty(storedState)) {
@@ -636,6 +651,10 @@ public class Profile {
return SummaryProfile.PROFILE_COMPLETION_STATE_UNKNOWN;
}
+ if (executionProfilesReleased) {
+ return SummaryProfile.PROFILE_COMPLETION_STATE_INCOMPLETE;
+ }
+
if (!isQueryFinished) {
return SummaryProfile.PROFILE_COMPLETION_STATE_RUNNING;
}
@@ -650,6 +669,9 @@ public class Profile {
}
private String getProfileCompletionStateForStorage() {
+ if (executionProfilesReleased) {
+ return SummaryProfile.PROFILE_COMPLETION_STATE_INCOMPLETE;
+ }
if (!isQueryFinished) {
return SummaryProfile.PROFILE_COMPLETION_STATE_RUNNING;
}
@@ -664,11 +686,18 @@ public class Profile {
}
private void updateProfileCompletionStateForStorage() {
-
summaryProfile.setProfileCompletionState(getProfileCompletionStateForStorage());
+ synchronized (completionStateLock) {
+ // Freeze the terminal state before serialization. Concurrent
rendering must not
+ // replace INCOMPLETE with COLLECTING while the storage path is
still unpublished.
+ storageProfileCompletionState =
getProfileCompletionStateForStorage();
+
summaryProfile.setProfileCompletionState(storageProfileCompletionState);
+ }
}
private void updateProfileCompletionStateForDisplay() {
- summaryProfile.setProfileCompletionState(getProfileCompletionState());
+ synchronized (completionStateLock) {
+
summaryProfile.setProfileCompletionState(getProfileCompletionState());
+ }
}
// Profile IO threads races with Coordinator threads.
@@ -741,6 +770,7 @@ public class Profile {
this.profileStoragePath = profileFilePath;
} catch (Exception e) {
+ storageProfileCompletionState = null;
LOG.error("write {} summary profile failed", getId(), e);
return;
} finally {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileCompletionStateTest.java
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileCompletionStateTest.java
new file mode 100644
index 00000000000..497e3e8f1d0
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileCompletionStateTest.java
@@ -0,0 +1,244 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.profile;
+
+import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.planner.PlanFragmentId;
+import org.apache.doris.thrift.TDetailedReportParams;
+import org.apache.doris.thrift.TNetworkAddress;
+import org.apache.doris.thrift.TQueryProfile;
+import org.apache.doris.thrift.TRuntimeProfileNode;
+import org.apache.doris.thrift.TRuntimeProfileTree;
+import org.apache.doris.thrift.TUniqueId;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.DataOutput;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+
+class ProfileCompletionStateTest {
+ @TempDir
+ Path temporary;
+
+ private String newFolder() throws IOException {
+ return Files.createTempDirectory(temporary, "profile_").toString();
+ }
+
+ private Profile profile() {
+ UUID id = UUID.randomUUID();
+ TUniqueId queryId = new TUniqueId(id.getMostSignificantBits(),
id.getLeastSignificantBits());
+ Profile profile = new Profile();
+
profile.getSummaryProfile().getSummary().addInfoString(SummaryProfile.PROFILE_ID,
DebugUtil.printId(queryId));
+ profile.addExecutionProfile(new ExecutionProfile(queryId,
Collections.singletonList(0)));
+ return profile;
+ }
+
+ private void report(ExecutionProfile execution, String host, boolean done)
{
+ TRuntimeProfileNode node = new TRuntimeProfileNode();
+ node.setName("PipelineProfile");
+ node.setNumChildren(0);
+ node.setCounters(new ArrayList<>());
+ node.setMetadata(0);
+ node.setIndent(false);
+ node.setInfoStrings(new HashMap<>());
+ node.setInfoStringsDisplayOrder(new ArrayList<>());
+ node.setChildCountersMap(new HashMap<>());
+ node.setTimestamp(0);
+ TDetailedReportParams params = new TDetailedReportParams();
+ params.setProfile(new
TRuntimeProfileTree(Collections.singletonList(node)));
+ params.setIsFragmentLevel(false);
+ TQueryProfile report = new TQueryProfile();
+ report.setQueryId(execution.getQueryId());
+ // Multiple pipeline nodes from one BE must not count as multiple BE
reports.
+ report.putToFragmentIdToProfile(0, Arrays.asList(params, params));
+ execution.updateProfile(report, new TNetworkAddress(host, 9050), done);
+ }
+
+ @Test
+ void waitsForEveryBackendAndRejectsStaleReports() {
+ Profile profile = profile();
+ ExecutionProfile execution = profile.getExecutionProfiles().get(0);
+ execution.addFragmentBackend(new PlanFragmentId(0), 1L);
+ execution.addFragmentBackend(new PlanFragmentId(0), 2L);
+ Assertions.assertEquals("RUNNING",
profile.getProfileCompletionState());
+ profile.markQueryFinished();
+ Assertions.assertEquals("COLLECTING",
profile.getProfileCompletionState());
+ report(execution, "127.0.0.1", true);
+ Assertions.assertEquals("COLLECTING",
profile.getProfileCompletionState());
+ report(execution, "127.0.0.2", true);
+ Assertions.assertEquals("COMPLETE",
profile.getProfileCompletionState());
+ report(execution, "127.0.0.1", false);
+ Assertions.assertEquals("COMPLETE",
profile.getProfileCompletionState());
+ Assertions.assertTrue(profile.getProfileByLevel().contains("Profile
Completion State: COMPLETE"));
+ }
+
+ private static class PausingSummaryProfile extends SummaryProfile {
+ private final transient boolean beforeSerialization;
+ private final transient CountDownLatch storagePaused;
+ private final transient CountDownLatch resumeStorage;
+
+ private PausingSummaryProfile(boolean beforeSerialization,
CountDownLatch storagePaused,
+ CountDownLatch resumeStorage) {
+ this.beforeSerialization = beforeSerialization;
+ this.storagePaused = storagePaused;
+ this.resumeStorage = resumeStorage;
+ }
+
+ @Override
+ public void write(DataOutput output) throws IOException {
+ if (!beforeSerialization) {
+ super.write(output);
+ }
+ storagePaused.countDown();
+ try {
+ if (!resumeStorage.await(10, TimeUnit.SECONDS)) {
+ throw new IOException("Timed out waiting for concurrent
profile rendering");
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new IOException(e);
+ }
+ if (beforeSerialization) {
+ super.write(output);
+ }
+ }
+ }
+
+ private static class FailingSummaryProfile extends SummaryProfile {
+ private transient boolean fail = true;
+
+ @Override
+ public void write(DataOutput output) throws IOException {
+ if (fail) {
+ fail = false;
+ throw new IOException("Injected storage failure");
+ }
+ super.write(output);
+ }
+ }
+
+ @Test
+ void failedStorageDoesNotFreezeCompletionState() throws Exception {
+ Profile profile = profile();
+ SummaryProfile summary = new FailingSummaryProfile();
+ summary.getSummary().addInfoString(SummaryProfile.PROFILE_ID,
profile.getId());
+ profile.setSummaryProfile(summary);
+ ExecutionProfile execution = profile.getExecutionProfiles().get(0);
+ execution.addFragmentBackend(new PlanFragmentId(0), 1L);
+ profile.markQueryFinished();
+ String directory = newFolder();
+ profile.writeToStorage(directory);
+ Assertions.assertFalse(profile.profileHasBeenStored());
+ Assertions.assertEquals("COLLECTING",
profile.getProfileCompletionState());
+ report(execution, "127.0.0.1", true);
+ profile.writeToStorage(directory);
+ Assertions.assertTrue(profile.profileHasBeenStored());
+ Assertions.assertEquals("COMPLETE",
Profile.read(profile.getProfileStoragePath()).getProfileCompletionState());
+ }
+
+ @Test
+ void renderingDuringStoragePreservesTerminalState() throws Exception {
+ for (boolean beforeSerialization : new boolean[] {true, false}) {
+ Profile profile = profile();
+ CountDownLatch storagePaused = new CountDownLatch(1);
+ CountDownLatch resumeStorage = new CountDownLatch(1);
+ SummaryProfile summary = new
PausingSummaryProfile(beforeSerialization, storagePaused, resumeStorage);
+ summary.getSummary().addInfoString(SummaryProfile.PROFILE_ID,
profile.getId());
+ profile.setSummaryProfile(summary);
+ profile.getExecutionProfiles().get(0).addFragmentBackend(new
PlanFragmentId(0), 1L);
+ profile.markQueryFinished();
+ String directory = newFolder();
+ ExecutorService writer = Executors.newSingleThreadExecutor();
+ try {
+ Future<?> stored = writer.submit(() ->
profile.writeToStorage(directory));
+ Assertions.assertTrue(storagePaused.await(10,
TimeUnit.SECONDS));
+ // Render both before serialization and before path
publication, while storage is paused.
+ profile.getProfileByLevel();
+ resumeStorage.countDown();
+ stored.get(10, TimeUnit.SECONDS);
+ Assertions.assertTrue(profile.profileHasBeenStored());
+ Assertions.assertEquals("INCOMPLETE",
profile.getProfileCompletionState());
+ Profile restored =
Profile.read(profile.getProfileStoragePath());
+ Assertions.assertNotNull(restored);
+ Assertions.assertEquals("INCOMPLETE",
restored.getProfileCompletionState());
+ } finally {
+ resumeStorage.countDown();
+ writer.shutdownNow();
+ Assertions.assertTrue(writer.awaitTermination(10,
TimeUnit.SECONDS));
+ }
+ }
+ }
+
+ @Test
+ void renderingDoesNotWaitForProfileMonitor() throws Exception {
+ Profile profile = profile();
+ ExecutionProfile execution = profile.getExecutionProfiles().get(0);
+ execution.addFragmentBackend(new PlanFragmentId(0), 1L);
+ profile.markQueryFinished();
+ report(execution, "127.0.0.1", true);
+ ExecutorService renderer = Executors.newSingleThreadExecutor();
+ try {
+ // updateSummary() holds this monitor while it takes
ProfileManager's write lock, and
+ // ProfileManager renders under its read lock, so rendering must
not need the monitor.
+ synchronized (profile) {
+ Future<String> rendered =
renderer.submit(profile::getProfileByLevel);
+ Assertions.assertTrue(rendered.get(10, TimeUnit.SECONDS)
+ .contains("Profile Completion State: COMPLETE"));
+ }
+ } finally {
+ renderer.shutdownNow();
+ Assertions.assertTrue(renderer.awaitTermination(10,
TimeUnit.SECONDS));
+ }
+ }
+
+ @Test
+ void persistsTerminalStateBeforeReleasingReports() throws Exception {
+ for (boolean complete : new boolean[] {false, true}) {
+ Profile profile = profile();
+ ExecutionProfile execution = profile.getExecutionProfiles().get(0);
+ execution.addFragmentBackend(new PlanFragmentId(0), 1L);
+ profile.markQueryFinished();
+ if (complete) {
+ report(execution, "127.0.0.1", true);
+ }
+ profile.writeToStorage(newFolder());
+ Assertions.assertTrue(profile.profileHasBeenStored());
+ profile.releaseMemory();
+ String expected = complete ? "COMPLETE" : "INCOMPLETE";
+ Assertions.assertEquals(expected,
profile.getProfileCompletionState());
+ Profile restored = Profile.read(profile.getProfileStoragePath());
+ Assertions.assertNotNull(restored);
+ Assertions.assertEquals(expected,
restored.getProfileCompletionState());
+
Assertions.assertTrue(restored.getProfileByLevel().contains("Profile Completion
State: " + expected));
+ }
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileManagerTest.java
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileManagerTest.java
index 75a7c39703c..38d1a4deb5d 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileManagerTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/ProfileManagerTest.java
@@ -571,6 +571,35 @@ class ProfileManagerTest {
Assertions.assertEquals(30 / 2, profiles.size());
}
+ @Test
+ void testFailedSpillRemainsIncompleteAfterMemoryRelease() throws Exception
{
+ UUID id = UUID.randomUUID();
+ TUniqueId queryId = new TUniqueId(id.getMostSignificantBits(),
id.getLeastSignificantBits());
+ Profile profile = constructProfile(DebugUtil.printId(queryId));
+ ExecutionProfile execution = new ExecutionProfile(queryId,
Lists.newArrayList(0));
+ execution.addFragmentBackend(0, 1L);
+ profile.addExecutionProfile(execution);
+ profile.markQueryFinished();
+ profile.setQueryFinishTimestamp(0);
+ profileManager.pushProfile(profile);
+ profileManager.addExecutionProfile(execution);
+ File invalidDirectory = new File(tempDir, "not_a_directory");
+ Assertions.assertTrue(invalidDirectory.createNewFile());
+ ProfileManager.PROFILE_STORAGE_PATH =
invalidDirectory.getAbsolutePath();
+
+ profileManager.writeProfileToStorage();
+
+ Assertions.assertFalse(profile.profileHasBeenStored());
+ Assertions.assertTrue(profile.getExecutionProfiles().isEmpty());
+ Assertions.assertEquals("INCOMPLETE",
profile.getProfileCompletionState());
+ Assertions.assertTrue(profile.getProfileByLevel().contains("Profile
Completion State: INCOMPLETE"));
+ // A later successful spill cannot recreate the reports discarded
after the failed write.
+ ProfileManager.PROFILE_STORAGE_PATH = tempDir.getAbsolutePath();
+ profileManager.writeProfileToStorage();
+ Assertions.assertTrue(profile.profileHasBeenStored());
+ Assertions.assertEquals("INCOMPLETE",
Profile.read(profile.getProfileStoragePath()).getProfileCompletionState());
+ }
+
@Test
void testWriteProfileToStorage() throws InterruptedException {
List<Profile> spyProfiles = new ArrayList<>();
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
