[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use

2026-10-06 Thread AsterixDB Code Review
>From Murtadha Hubail :

Murtadha Hubail has abandoned this change. ( 
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email )

Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in 
use
..


Abandoned
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email
To unsubscribe, or for help writing mail filters, visit 
https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: abandon
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf
Gerrit-Change-Number: 21959
Gerrit-PatchSet: 1
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 


[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use

2026-10-04 Thread AsterixDB Code Review
>From Murtadha Hubail :

Attention is currently required from: Ian Maxon.

Murtadha Hubail has posted comments on this change by Murtadha Hubail. ( 
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email )

Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in 
use
..


Patch Set 1:

(1 comment)

Patchset:

PS1:
> the specific reason is that it is making the traversal code more permissive 
> (i.e. […]
I think we both agree that we should go the Metadata catalog route. For the 
record, the listing API takes no user input, and I believe the patch doesn't 
widen what the walk can reach or what the listing can return. So it shouldn't 
introduce any path traversal or data-exposure issue that wasn't already there. 
It also meets both conditions you stated:
- Only the expected storage pattern is traversed. The walk starts at the fixed 
storageDirPath and doesn't follow symlinks. Every folder that doesn't match 
[db/]dv/[^dv2/...]/lib gets SKIP_SUBTREE.
- Only expected names are returned. A name comes only from a folder that 
matches that pattern and contains a revision folder.

The patch also has a unit test covering the database (cloud) and non-database 
layouts, multi-part dataverses, folders that aren't libraries, and stray files.

As discussed, let's start by disabling this API and go the Metadata catalog 
route when we revisit it.



--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email
To unsubscribe, or for help writing mail filters, visit 
https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: comment
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf
Gerrit-Change-Number: 21959
Gerrit-PatchSet: 1
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Attention: Ian Maxon 
Gerrit-Comment-Date: Sun, 04 Oct 2026 13:19:53 +
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Comment-In-Reply-To: Murtadha Hubail 
Comment-In-Reply-To: Ian Maxon 


[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use

2026-10-02 Thread AsterixDB Code Review
>From Ian Maxon :

Attention is currently required from: Murtadha Hubail.

Ian Maxon has posted comments on this change by Murtadha Hubail. ( 
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email )

Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in 
use
..


Patch Set 1:

(1 comment)

Patchset:

PS1:
assuming https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21977 is fine, this 
will fix it and improve things from a security standpoint. the only issue i see 
is that it still gives the old format which only gives dataverse, and not 
database, as a top-level response in the query. it uses . to catenate the two 
together into a fully-qualified dataverse. however it does fix it.



--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email
To unsubscribe, or for help writing mail filters, visit 
https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: comment
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf
Gerrit-Change-Number: 21959
Gerrit-PatchSet: 1
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Attention: Murtadha Hubail 
Gerrit-Comment-Date: Fri, 02 Oct 2026 18:23:20 +
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No


[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use

2026-10-02 Thread AsterixDB Code Review
>From Ian Maxon :

Attention is currently required from: Murtadha Hubail.

Ian Maxon has posted comments on this change by Murtadha Hubail. ( 
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email )

Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in 
use
..


Patch Set 1:

(1 comment)

Patchset:

PS1:
i will try to propose a patch with the alternate method we both agree on 
(asking Metadata) in a bit, i just need to give it a look.



--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email
To unsubscribe, or for help writing mail filters, visit 
https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: comment
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf
Gerrit-Change-Number: 21959
Gerrit-PatchSet: 1
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Attention: Murtadha Hubail 
Gerrit-Comment-Date: Fri, 02 Oct 2026 16:54:34 +
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No


[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use

2026-10-02 Thread AsterixDB Code Review
>From Ian Maxon :

Attention is currently required from: Murtadha Hubail.

Ian Maxon has posted comments on this change by Murtadha Hubail. ( 
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email )

Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in 
use
..


Patch Set 1: Code-Review-2

(1 comment)

Patchset:

PS1:
> Any specific reason? Doesn't this patch make the existing API work for both 
> cloud and non-cloud mode […]
the specific reason is that it is making the traversal code more permissive 
(i.e. allowing CONTINUE where currently any file leads to TERMINATE). this is 
dangerous, and i strongly advise against doing it without careful inspection. i 
don't disagree with fixing it eventually, but it cannot be something claude 
oneshots (with no tests) and i give a terse look to. this is intricate code 
which if it does not:
 - only allow the *exact* file pattern we expect from storage to be traversed
 - only allows the names of the paths we expect to be components of the library 
name to be retrieved

can lead to a critical security vulnerability. because it has before. if you 
want to merge it i can't stop you, but don't say i didn't say anything.

i agree that it should be done a different way (e.g. simply proxying metadata). 
that is further evidence to just leave it be for now. the way it's done here is 
not a good idea in general, and if you do it this way, you have to be 
exceedingly careful.



--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email
To unsubscribe, or for help writing mail filters, visit 
https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: comment
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf
Gerrit-Change-Number: 21959
Gerrit-PatchSet: 1
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Attention: Murtadha Hubail 
Gerrit-Comment-Date: Fri, 02 Oct 2026 14:43:23 +
Gerrit-HasComments: Yes
Gerrit-Has-Labels: Yes
Comment-In-Reply-To: Murtadha Hubail 
Comment-In-Reply-To: Ian Maxon 


[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use

2026-10-02 Thread AsterixDB Code Review
>From Murtadha Hubail :

Attention is currently required from: Ian Maxon.

Murtadha Hubail has posted comments on this change by Murtadha Hubail. ( 
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email )

Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in 
use
..


Patch Set 1: Code-Review+1 Integration-Tests+1 Verified+1


--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email
To unsubscribe, or for help writing mail filters, visit 
https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: comment
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf
Gerrit-Change-Number: 21959
Gerrit-PatchSet: 1
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Attention: Ian Maxon 
Gerrit-Comment-Date: Fri, 02 Oct 2026 11:42:20 +
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes


[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use

2026-10-02 Thread AsterixDB Code Review
>From Murtadha Hubail :

Attention is currently required from: Ian Maxon.

Murtadha Hubail has posted comments on this change by Murtadha Hubail. ( 
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email )

Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in 
use
..


Patch Set 1:

(1 comment)

Patchset:

PS1:
> i would prefer to do something like https://asterix-gerrit.ics.uci. […]
Any specific reason? Doesn't this patch make the existing API work for both 
cloud and non-cloud modes with a smaller diff? Also, if the authoritative list 
is supposed to be whatever is in Library metadata dataset, then I wonder if 
this API should just proxy to the metadata node rather than traversing storage, 
but that can potentially be done later without changing the user facing API.



--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email
To unsubscribe, or for help writing mail filters, visit 
https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: comment
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf
Gerrit-Change-Number: 21959
Gerrit-PatchSet: 1
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Attention: Ian Maxon 
Gerrit-Comment-Date: Fri, 02 Oct 2026 10:51:50 +
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Comment-In-Reply-To: Ian Maxon 


[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use

2026-10-02 Thread AsterixDB Code Review
>From Murtadha Hubail :

Attention is currently required from: Ian Maxon.

Murtadha Hubail has removed a vote from this change. ( 
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email )


Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in 
use
..


Removed Integration-Tests-1 by Jenkins 
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email
To unsubscribe, or for help writing mail filters, visit 
https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: deleteVote
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf
Gerrit-Change-Number: 21959
Gerrit-PatchSet: 1
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Attention: Ian Maxon 


[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use

2026-10-02 Thread AsterixDB Code Review
>From Murtadha Hubail :

Attention is currently required from: Ian Maxon.

Murtadha Hubail has removed a vote from this change. ( 
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email )


Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in 
use
..


Removed Verified-1 by Jenkins 
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email
To unsubscribe, or for help writing mail filters, visit 
https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: deleteVote
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf
Gerrit-Change-Number: 21959
Gerrit-PatchSet: 1
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-Attention: Ian Maxon 


[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use

2026-10-01 Thread AsterixDB Code Review
>From Ian Maxon :

Attention is currently required from: Murtadha Hubail.

Ian Maxon has posted comments on this change by Murtadha Hubail. ( 
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email )

Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in 
use
..


Patch Set 1:

(1 comment)

Patchset:

PS1:
i would prefer to do something like 
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21961 instead



--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email
To unsubscribe, or for help writing mail filters, visit 
https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: comment
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf
Gerrit-Change-Number: 21959
Gerrit-PatchSet: 1
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Ian Maxon 
Gerrit-Reviewer: Jenkins 
Gerrit-Attention: Murtadha Hubail 
Gerrit-Comment-Date: Thu, 01 Oct 2026 19:49:56 +
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No


[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use

2026-10-01 Thread AsterixDB Code Review
>From Jenkins :

Attention is currently required from: Murtadha Hubail.

Jenkins has posted comments on this change by Murtadha Hubail. ( 
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email )

Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in 
use
..


Patch Set 1: Integration-Tests-1

Integration Tests Failed

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/8024/ : UNSTABLE


--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email
To unsubscribe, or for help writing mail filters, visit 
https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: comment
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf
Gerrit-Change-Number: 21959
Gerrit-PatchSet: 1
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-Attention: Murtadha Hubail 
Gerrit-Comment-Date: Thu, 01 Oct 2026 18:00:44 +
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes


[M] Change in asterixdb[master]: [NO ISSUE][EXT] Fix the library listing when databases are in use

2026-10-01 Thread AsterixDB Code Review
>From Murtadha Hubail :

Murtadha Hubail has uploaded this change for review. ( 
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21959?usp=email )


Change subject: [NO ISSUE][EXT] Fix the library listing when databases are in 
use
..

[NO ISSUE][EXT] Fix the library listing when databases are in use

With databases, library storage is laid out as db/dv/lib. The listing
walk returned TERMINATE at the db/dv directory, so the whole listing
came back empty, and the branch that was meant to handle databases also
indexed past the end of the path and ignored the database name.

Ext-ref: MB-67950
Generated-by: Claude Opus 5.5 
Change-Id: Ic0300108f9ee051415b68b05932dab6effc1dbaf
---
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalLibraryManager.java
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/ExternalLibraryListingTest.java
2 files changed, 113 insertions(+), 20 deletions(-)



  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/59/21959/1

diff --git 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalLibraryManager.java
 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalLibraryManager.java
index 3841146..b8a91f7 100755
--- 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalLibraryManager.java
+++ 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/ExternalLibraryManager.java
@@ -275,18 +275,24 @@

 @Override
 public List> getLibraryListing() throws 
IOException {
+return listLibraries(storageDirPath, 
namespacePathResolver.usingDatabase());
+}
+
+static List> listLibraries(Path storageDirPath, 
boolean usingDatabase) throws IOException {
 List> libs = new ArrayList<>();
+// with databases, the layout is db/dv/^dv2/.../lib rather than 
dv/^dv2/.../lib
+int dvStartToken = usingDatabase ? 1 : 0;
 Files.walkFileTree(storageDirPath, new SimpleFileVisitor() {
 @Override
 public FileVisitResult visitFile(Path currPath, 
BasicFileAttributes attrs) {
-//never want to see any files
-return FileVisitResult.TERMINATE;
+// a stray file must not cut the listing short
+return FileVisitResult.CONTINUE;
 }

 @Override
 public FileVisitResult preVisitDirectory(Path currPath, 
BasicFileAttributes attrs)
 throws HyracksDataException {
-if (currPath.equals(storageDirPath) || 
currPath.getParent().equals(storageDirPath)) {
+if (currPath.equals(storageDirPath)) {
 return FileVisitResult.CONTINUE;
 }
 if (currPath.getFileName().toString().codePointAt(0) == 
StoragePathUtil.DATAVERSE_CONTINUATION_MARKER) {
@@ -296,24 +302,15 @@
 
.relativize(currPath.toAbsolutePath().normalize()).toString();
 List dvParts = new ArrayList<>();
 final String[] tokens = StringUtils.split(candidateDvAndLib, 
File.separatorChar);
-if (tokens == null || tokens.length < 2) {
-//? shouldn't happen
-return FileVisitResult.TERMINATE;
+if (tokens.length < dvStartToken + 2) {
+// a database or dataverse directory; libraries are 
further down
+return FileVisitResult.CONTINUE;
 }
 try {
-String candidateDb = MetadataConstants.DEFAULT_DATABASE;
-if (namespacePathResolver.usingDatabase()) {
-if (tokens.length < 3) {
-return FileVisitResult.TERMINATE;
-}
-libs.add(Pair.of(new Namespace(candidateDb, 
DataverseName.create(List.of(tokens[1]))),
-tokens[3]));
-return FileVisitResult.SKIP_SUBTREE;
-
-}
+String candidateDb = usingDatabase ? tokens[0] : 
MetadataConstants.DEFAULT_DATABASE;
 //add first part, then look for multiparts
-dvParts.add(tokens[0]);
-int currToken = 1;
+dvParts.add(tokens[dvStartToken]);
+int currToken = dvStartToken + 1;
 for (; currToken < tokens.length && tokens[currToken]
 .codePointAt(0) == 
StoragePathUtil.DATAVERSE_CONTINUATION_MARKER; currToken++) {
 dvParts.add(tokens[currToken].substring(1));
@@ -327,8 +324,8 @@
 DataverseName candidateDv;
 candidateDv = DataverseName.create(dvParts);