nick-boss-tech commented on code in PR #5352:
URL: https://github.com/apache/solr/pull/5352#discussion_r4636871553
##########
solr/core/src/test/org/apache/solr/cli/PackageToolTest.java:
##########
@@ -300,6 +307,41 @@ public void testPackageTool() throws Exception {
"--credentials",
SecurityJson.USER_PASS
});
+
+ var packageRequest = new PackageApi.GetPackage("question-answer");
+ packageRequest.setBasicAuthCredentials(SecurityJson.USER,
SecurityJson.PASS);
+ var versions = packageRequest.process(cluster.getSolrClient()).versions;
+ assertEquals(2, versions.size());
+ for (var version : versions) {
+ List<String> files = new ArrayList<>(version.files);
+ files.add(version.manifest);
+ for (String file : files) {
+ assertTrue(cluster.getZkClient().exists("/packagestore" + file));
+ }
+
+ run(
+ new String[] {
+ "--solr-url",
+ solrUrl,
+ "uninstall",
Review Comment:
🤖 *AI text below* 🤖
Could we cover an uninstall that gets past package removal but fails during
file cleanup? [The CLI removes the package version
first](https://github.com/epugh/solr/blob/35e418738b28d91d5c70722e09ed9b0a476d48b1/solr/core/src/java/org/apache/solr/cli/packagemanager/PackageManager.java#L147-L166),
then stops at the first failed file DELETE. A retry now can’t look up the
version to recover its file list. This is especially likely if the new CLI
reaches an older server, since the CLI no longer removes the ZooKeeper entry
and that server still rejects registered-file deletes. Could we check
compatibility before unregistering and make remaining cleanup resumable after a
partial failure?
##########
solr/solr-ref-guide/modules/configuration-guide/pages/package-manager-internals.adoc:
##########
@@ -90,6 +90,7 @@ The end points are:
* `PUT /api/cluster/filestore/files/{full/path/to/file}` to upload a new file
and add it to the file store
* `GET /api/cluster/filestore/files/{full/path/to/file}` to download a file
already in the filestore
+* `DELETE /api/cluster/filestore/files/{full/path/to/file}` to remove a file
from ZooKeeper and delete its copies on live nodes. Files used by an installed
package cannot be deleted.
Review Comment:
🤖 *AI text below* 🤖
Could the in-use check cover manifests as well as artifacts before this
endpoint starts removing registered files?
[`isJarInuse()`](https://github.com/epugh/solr/blob/35e418738b28d91d5c70722e09ed9b0a476d48b1/solr/core/src/java/org/apache/solr/pkg/PackageStore.java#L375-L391)
only checks `version.files`, but the installer [registers the manifest
separately](https://github.com/epugh/solr/blob/35e418738b28d91d5c70722e09ed9b0a476d48b1/solr/core/src/java/org/apache/solr/cli/packagemanager/RepositoryManager.java#L209-L223).
Package management still reads that manifest for installed versions. Please
keep both kinds of references protected and add a regression assertion for the
manifest.
##########
solr/core/src/test/org/apache/solr/filestore/TestDistribFileStore.java:
##########
@@ -193,6 +194,25 @@ public void testFileStoreManagement() throws Exception {
expected =
Collections.singletonMap(":files:/package/mypkg/v1.0/runtimelibs.jar", null);
checkAllNodesForFile(cluster, "/package/mypkg/v1.0/runtimelibs.jar",
expected, false);
+
+ final String filePath = "/package/mypkg/v1.0/runtimelibs_v2.jar";
+ final String fileUrl = j.getBaseURLV2() + "/cluster/filestore/files" +
filePath;
+ assertTrue(cluster.getZkClient().exists(DistribFileStore.ZK_PACKAGESTORE
+ filePath));
+
+ resp =
+ j.getSolrClient()
+ .getHttpClient()
+ .newRequest(fileUrl + "?localDelete=true")
+ .method("DELETE")
+ .send();
+ assertEquals(400, resp.getStatus());
+ assertTrue(cluster.getZkClient().exists(DistribFileStore.ZK_PACKAGESTORE
+ filePath));
+
+ resp =
j.getSolrClient().getHttpClient().newRequest(fileUrl).method("DELETE").send();
Review Comment:
🤖 *AI text below* 🤖
Could we cover pending file transfers when asserting that cluster DELETE is
complete? [A fetch verifies the downloaded data and then persists
it](https://github.com/epugh/solr/blob/35e418738b28d91d5c70722e09ed9b0a476d48b1/solr/core/src/java/org/apache/solr/filestore/DistribFileStore.java#L193-L214)
without checking that the same registration is still current. It can restore a
deleted copy even if every delete succeeded. Conversely, a delayed delete
shouldn’t touch a replacement at the same path. Binding those operations to a
registration generation and coordinating that check with the local write would
protect both cases.
##########
solr/core/src/test/org/apache/solr/filestore/TestDistribFileStore.java:
##########
@@ -193,6 +194,25 @@ public void testFileStoreManagement() throws Exception {
expected =
Collections.singletonMap(":files:/package/mypkg/v1.0/runtimelibs.jar", null);
checkAllNodesForFile(cluster, "/package/mypkg/v1.0/runtimelibs.jar",
expected, false);
+
+ final String filePath = "/package/mypkg/v1.0/runtimelibs_v2.jar";
+ final String fileUrl = j.getBaseURLV2() + "/cluster/filestore/files" +
filePath;
+ assertTrue(cluster.getZkClient().exists(DistribFileStore.ZK_PACKAGESTORE
+ filePath));
+
+ resp =
+ j.getSolrClient()
+ .getHttpClient()
+ .newRequest(fileUrl + "?localDelete=true")
+ .method("DELETE")
+ .send();
+ assertEquals(400, resp.getStatus());
+ assertTrue(cluster.getZkClient().exists(DistribFileStore.ZK_PACKAGESTORE
+ filePath));
+
+ resp =
j.getSolrClient().getHttpClient().newRequest(fileUrl).method("DELETE").send();
+ assertEquals(200, resp.getStatus());
+
assertFalse(cluster.getZkClient().exists(DistribFileStore.ZK_PACKAGESTORE +
filePath));
Review Comment:
🤖 *AI text below* 🤖
Could we make the file-in-use decision atomic with package registration?
[Delete checks references before its write
lock](https://github.com/epugh/solr/blob/35e418738b28d91d5c70722e09ed9b0a476d48b1/solr/core/src/java/org/apache/solr/filestore/ClusterFileStore.java#L290-L328),
and [package registration validates the files before recording the
version](https://github.com/epugh/solr/blob/35e418738b28d91d5c70722e09ed9b0a476d48b1/solr/core/src/java/org/apache/solr/pkg/PackageStore.java#L256-L291)
without taking that lock. Now that DELETE can remove registered files,
overlapping operations can leave an installed version pointing at a missing
artifact. Rechecking under a lock shared by both operations, or using a
transaction that checks both states, would preserve that invariant.
##########
solr/core/src/test/org/apache/solr/filestore/TestDistribFileStore.java:
##########
@@ -193,6 +194,25 @@ public void testFileStoreManagement() throws Exception {
expected =
Collections.singletonMap(":files:/package/mypkg/v1.0/runtimelibs.jar", null);
checkAllNodesForFile(cluster, "/package/mypkg/v1.0/runtimelibs.jar",
expected, false);
+
+ final String filePath = "/package/mypkg/v1.0/runtimelibs_v2.jar";
+ final String fileUrl = j.getBaseURLV2() + "/cluster/filestore/files" +
filePath;
+ assertTrue(cluster.getZkClient().exists(DistribFileStore.ZK_PACKAGESTORE
+ filePath));
+
+ resp =
+ j.getSolrClient()
+ .getHttpClient()
+ .newRequest(fileUrl + "?localDelete=true")
+ .method("DELETE")
+ .send();
+ assertEquals(400, resp.getStatus());
+ assertTrue(cluster.getZkClient().exists(DistribFileStore.ZK_PACKAGESTORE
+ filePath));
+
+ resp =
j.getSolrClient().getHttpClient().newRequest(fileUrl).method("DELETE").send();
+ assertEquals(200, resp.getStatus());
Review Comment:
🤖 *AI text below* 🤖
Could we cover a remote delete failure here? The success case waits for
copies to disappear, but [the server ignores the remote
futures](https://github.com/epugh/solr/blob/35e418738b28d91d5c70722e09ed9b0a476d48b1/solr/core/src/java/org/apache/solr/filestore/DistribFileStore.java#L495-L508).
If a live node fails its delete, the API can still return 200 and remove the
ZooKeeper entry while that node retains a copy. If cluster deletion is best
effort, say so in the guide. If 200 means all live copies are gone, wait for
the results and surface failures so callers can retry.
##########
solr/core/src/java/org/apache/solr/filestore/DistribFileStore.java:
##########
@@ -486,6 +485,7 @@ public List<FileDetails> list(String path,
Predicate<String> predicate) {
@Override
public void delete(String path) {
+ deleteZKFileEntry(path);
Review Comment:
🤖 *AI text below* 🤖
Could cluster DELETE treat a registered file as present even if no live node
has its bytes?
[`doClusterDelete()`](https://github.com/epugh/solr/blob/35e418738b28d91d5c70722e09ed9b0a476d48b1/solr/core/src/java/org/apache/solr/filestore/ClusterFileStore.java#L283-L295)
currently calls `getType(path, true)` and returns 400 on `NOFILE` before it
reaches this line. That leaves stale ZooKeeper registrations in place. The old
package CLI could clear the registration on its own; after this change it
depends on the endpoint. Deleting a registered file shouldn’t require
downloading it first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]