>From Ian Maxon <[email protected]>: Attention is currently required from: Murtadha Hubail, Peeyush Gupta, Ritik Raj. Ian Maxon has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18774 )
Change subject: [ASTERIXDB-3495] Library Cloud Storage ...................................................................... Patch Set 16: (2 comments) File asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/AbstractCloudIOManager.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18774/comment/b8871305_be8b2960 PS16, Line 198: public void downloadAllLibraries() throws HyracksDataException { : IParallelDownloader downloader = cloudClient.createParallelDownloader(bucket, localIoManager); : FileReference appDir = resolveAbsolutePath( : localIoManager.getWorkspacePath(0).getPath() + File.separator + APPLICATION_ROOT_DIR_NAME); : LOGGER.info("Downloading all libraries in + {}", appDir); : downloader.downloadDirectories(Collections.singletonList(appDir)); : LOGGER.info("Finished downloading all libraries"); : } : > Can the library files be large? If yes, then wouldn't it better to download > them lazily if the lazy […] yes, but i'd like to defer it for now if that is fine with you. it would have to happen in a different way than most of the lazy storage works, because the only way we know how to page them in or out is based on if a query attempts to load a library. it can't be done at the iomanager level because we don't control reads to it; the whole library has to be present before we attempt to use it. it would be most similar to how datasets are activated or deactivated. File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/operators/LibraryDeployPrepareOperatorDescriptor.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18774/comment/05c59494_0aaaefa5 PS16, Line 81: cloudIoManager.downloadLibrary(Collections.singletonList(libDir)); > Aren't we downloading all libraries at bootstrap? yes, but this is called when a new library is dropped into the bucket by the servlet so it can be extracted to the nvme -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18774 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I2c173fbdc985ea590da5315c1ce18e7610fb6af0 Gerrit-Change-Number: 18774 Gerrit-PatchSet: 16 Gerrit-Owner: Ian Maxon <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Murtadha Hubail <[email protected]> Gerrit-Reviewer: Peeyush Gupta <[email protected]> Gerrit-Reviewer: Ritik Raj <[email protected]> Gerrit-Attention: Murtadha Hubail <[email protected]> Gerrit-Attention: Peeyush Gupta <[email protected]> Gerrit-Attention: Ritik Raj <[email protected]> Gerrit-Comment-Date: Fri, 14 Feb 2025 01:46:23 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Comment-In-Reply-To: Peeyush Gupta <[email protected]> Gerrit-MessageType: comment
