[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use
>From Murtadha Hubail : Murtadha Hubail has abandoned this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email ) Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in use .. Abandoned -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: abandon Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf Gerrit-Change-Number: 21959 Gerrit-PatchSet: 1 Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail
[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use
>From Murtadha Hubail : Attention is currently required from: Ian Maxon. Murtadha Hubail has posted comments on this change by Murtadha Hubail. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email ) Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in use .. Patch Set 1: (1 comment) Patchset: PS1: > the specific reason is that it is making the traversal code more permissive > (i.e. […] I think we both agree that we should go the Metadata catalog route. For the record, the listing API takes no user input, and I believe the patch doesn't widen what the walk can reach or what the listing can return. So it shouldn't introduce any path traversal or data-exposure issue that wasn't already there. It also meets both conditions you stated: - Only the expected storage pattern is traversed. The walk starts at the fixed storageDirPath and doesn't follow symlinks. Every folder that doesn't match [db/]dv/[^dv2/...]/lib gets SKIP_SUBTREE. - Only expected names are returned. A name comes only from a folder that matches that pattern and contains a revision folder. The patch also has a unit test covering the database (cloud) and non-database layouts, multi-part dataverses, folders that aren't libraries, and stray files. As discussed, let's start by disabling this API and go the Metadata catalog route when we revisit it. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf Gerrit-Change-Number: 21959 Gerrit-PatchSet: 1 Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Attention: Ian Maxon Gerrit-Comment-Date: Sun, 04 Oct 2026 13:19:53 + Gerrit-HasComments: Yes Gerrit-Has-Labels: No Comment-In-Reply-To: Murtadha Hubail Comment-In-Reply-To: Ian Maxon
[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use
>From Ian Maxon : Attention is currently required from: Murtadha Hubail. Ian Maxon has posted comments on this change by Murtadha Hubail. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email ) Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in use .. Patch Set 1: (1 comment) Patchset: PS1: assuming https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21977 is fine, this will fix it and improve things from a security standpoint. the only issue i see is that it still gives the old format which only gives dataverse, and not database, as a top-level response in the query. it uses . to catenate the two together into a fully-qualified dataverse. however it does fix it. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf Gerrit-Change-Number: 21959 Gerrit-PatchSet: 1 Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Attention: Murtadha Hubail Gerrit-Comment-Date: Fri, 02 Oct 2026 18:23:20 + Gerrit-HasComments: Yes Gerrit-Has-Labels: No
[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use
>From Ian Maxon : Attention is currently required from: Murtadha Hubail. Ian Maxon has posted comments on this change by Murtadha Hubail. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email ) Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in use .. Patch Set 1: (1 comment) Patchset: PS1: i will try to propose a patch with the alternate method we both agree on (asking Metadata) in a bit, i just need to give it a look. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf Gerrit-Change-Number: 21959 Gerrit-PatchSet: 1 Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Attention: Murtadha Hubail Gerrit-Comment-Date: Fri, 02 Oct 2026 16:54:34 + Gerrit-HasComments: Yes Gerrit-Has-Labels: No
[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use
>From Ian Maxon : Attention is currently required from: Murtadha Hubail. Ian Maxon has posted comments on this change by Murtadha Hubail. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email ) Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in use .. Patch Set 1: Code-Review-2 (1 comment) Patchset: PS1: > Any specific reason? Doesn't this patch make the existing API work for both > cloud and non-cloud mode […] the specific reason is that it is making the traversal code more permissive (i.e. allowing CONTINUE where currently any file leads to TERMINATE). this is dangerous, and i strongly advise against doing it without careful inspection. i don't disagree with fixing it eventually, but it cannot be something claude oneshots (with no tests) and i give a terse look to. this is intricate code which if it does not: - only allow the *exact* file pattern we expect from storage to be traversed - only allows the names of the paths we expect to be components of the library name to be retrieved can lead to a critical security vulnerability. because it has before. if you want to merge it i can't stop you, but don't say i didn't say anything. i agree that it should be done a different way (e.g. simply proxying metadata). that is further evidence to just leave it be for now. the way it's done here is not a good idea in general, and if you do it this way, you have to be exceedingly careful. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf Gerrit-Change-Number: 21959 Gerrit-PatchSet: 1 Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Attention: Murtadha Hubail Gerrit-Comment-Date: Fri, 02 Oct 2026 14:43:23 + Gerrit-HasComments: Yes Gerrit-Has-Labels: Yes Comment-In-Reply-To: Murtadha Hubail Comment-In-Reply-To: Ian Maxon
[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use
>From Murtadha Hubail : Attention is currently required from: Ian Maxon. Murtadha Hubail has posted comments on this change by Murtadha Hubail. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email ) Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in use .. Patch Set 1: Code-Review+1 Integration-Tests+1 Verified+1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf Gerrit-Change-Number: 21959 Gerrit-PatchSet: 1 Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Attention: Ian Maxon Gerrit-Comment-Date: Fri, 02 Oct 2026 11:42:20 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes
[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use
>From Murtadha Hubail : Attention is currently required from: Ian Maxon. Murtadha Hubail has posted comments on this change by Murtadha Hubail. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email ) Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in use .. Patch Set 1: (1 comment) Patchset: PS1: > i would prefer to do something like https://asterix-gerrit.ics.uci. […] Any specific reason? Doesn't this patch make the existing API work for both cloud and non-cloud modes with a smaller diff? Also, if the authoritative list is supposed to be whatever is in Library metadata dataset, then I wonder if this API should just proxy to the metadata node rather than traversing storage, but that can potentially be done later without changing the user facing API. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf Gerrit-Change-Number: 21959 Gerrit-PatchSet: 1 Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Attention: Ian Maxon Gerrit-Comment-Date: Fri, 02 Oct 2026 10:51:50 + Gerrit-HasComments: Yes Gerrit-Has-Labels: No Comment-In-Reply-To: Ian Maxon
[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use
>From Murtadha Hubail : Attention is currently required from: Ian Maxon. Murtadha Hubail has removed a vote from this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email ) Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in use .. Removed Integration-Tests-1 by Jenkins -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: deleteVote Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf Gerrit-Change-Number: 21959 Gerrit-PatchSet: 1 Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Attention: Ian Maxon
[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use
>From Murtadha Hubail : Attention is currently required from: Ian Maxon. Murtadha Hubail has removed a vote from this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email ) Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in use .. Removed Verified-1 by Jenkins -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: deleteVote Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf Gerrit-Change-Number: 21959 Gerrit-PatchSet: 1 Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Attention: Ian Maxon
[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use
>From Ian Maxon : Attention is currently required from: Murtadha Hubail. Ian Maxon has posted comments on this change by Murtadha Hubail. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email ) Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in use .. Patch Set 1: (1 comment) Patchset: PS1: i would prefer to do something like https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21961 instead -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf Gerrit-Change-Number: 21959 Gerrit-PatchSet: 1 Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Attention: Murtadha Hubail Gerrit-Comment-Date: Thu, 01 Oct 2026 19:49:56 + Gerrit-HasComments: Yes Gerrit-Has-Labels: No
[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use
>From Jenkins : Attention is currently required from: Murtadha Hubail. Jenkins has posted comments on this change by Murtadha Hubail. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email ) Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in use .. Patch Set 1: Integration-Tests-1 Integration Tests Failed https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/8024/ : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: comment Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf Gerrit-Change-Number: 21959 Gerrit-PatchSet: 1 Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins Gerrit-Attention: Murtadha Hubail Gerrit-Comment-Date: Thu, 01 Oct 2026 18:00:44 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes
[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use
>From Murtadha Hubail :
Murtadha Hubail has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email )
Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in
use
..
[NO ISSUE][EXT] Fix the library listing when databases are in use
With databases, library storage is laid out as db/dv/lib. The listing
walk returned TERMINATE at the db/dv directory, so the whole listing
came back empty, and the branch that was meant to handle databases also
indexed past the end of the path and ignored the database name.
Ext-ref: MB-67950
Generated-by: Claude Opus 5.5
Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf
---
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalLibraryManager.java
A
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/ExternalLibraryListingTest.java
2 files changed, 113 insertions(+), 20 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/59/21959/1
diff --git
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalLibraryManager.java
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalLibraryManager.java
index 3841146..b8a91f7 100755
---
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalLibraryManager.java
+++
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalLibraryManager.java
@@ -275,18 +275,24 @@
@Override
public List> getLibraryListing() throws
IOException {
+return listLibraries(storageDirPath,
namespacePathResolver.usingDatabase());
+}
+
+static List> listLibraries(Path storageDirPath,
boolean usingDatabase) throws IOException {
List> libs = new ArrayList<>();
+// with databases, the layout is db/dv/^dv2/.../lib rather than
dv/^dv2/.../lib
+int dvStartToken = usingDatabase ? 1 : 0;
Files.walkFileTree(storageDirPath, new SimpleFileVisitor() {
@Override
public FileVisitResult visitFile(Path currPath,
BasicFileAttributes attrs) {
-//never want to see any files
-return FileVisitResult.TERMINATE;
+// a stray file must not cut the listing short
+return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult preVisitDirectory(Path currPath,
BasicFileAttributes attrs)
throws HyracksDataException {
-if (currPath.equals(storageDirPath) ||
currPath.getParent().equals(storageDirPath)) {
+if (currPath.equals(storageDirPath)) {
return FileVisitResult.CONTINUE;
}
if (currPath.getFileName().toString().codePointAt(0) ==
StoragePathUtil.DATAVERSE_CONTINUATION_MARKER) {
@@ -296,24 +302,15 @@
.relativize(currPath.toAbsolutePath().normalize()).toString();
List dvParts = new ArrayList<>();
final String[] tokens = StringUtils.split(candidateDvAndLib,
File.separatorChar);
-if (tokens == null || tokens.length < 2) {
-//? shouldn't happen
-return FileVisitResult.TERMINATE;
+if (tokens.length < dvStartToken + 2) {
+// a database or dataverse directory; libraries are
further down
+return FileVisitResult.CONTINUE;
}
try {
-String candidateDb = MetadataConstants.DEFAULT_DATABASE;
-if (namespacePathResolver.usingDatabase()) {
-if (tokens.length < 3) {
-return FileVisitResult.TERMINATE;
-}
-libs.add(Pair.of(new Namespace(candidateDb,
DataverseName.create(List.of(tokens[1]))),
-tokens[3]));
-return FileVisitResult.SKIP_SUBTREE;
-
-}
+String candidateDb = usingDatabase ? tokens[0] :
MetadataConstants.DEFAULT_DATABASE;
//add first part, then look for multiparts
-dvParts.add(tokens[0]);
-int currToken = 1;
+dvParts.add(tokens[dvStartToken]);
+int currToken = dvStartToken + 1;
for (; currToken < tokens.length && tokens[currToken]
.codePointAt(0) ==
StoragePathUtil.DATAVERSE_CONTINUATION_MARKER; currToken++) {
dvParts.add(tokens[currToken].substring(1));
@@ -327,8 +324,8 @@
DataverseName candidateDv;
candidateDv = DataverseName.create(dvParts);
