>From Ali Alsuliman <[email protected]>: Ali Alsuliman has submitted this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19954 )
Change subject: [ASTERIXDB-3623][OTH] Wait for submitted cloud requests on exceeding max pending HTTP connections ...................................................................... [ASTERIXDB-3623][OTH] Wait for submitted cloud requests on exceeding max pending HTTP connections - user model changes: no - storage format changes: no - interface changes: no Details: When downloading files in parallel, download up to max allowed pending, then wait for them to (acquire a connection and) finish. Continue with the rest. Ext-ref: MB-67243 Change-Id: Ifd1a30c6eed5f316a2d7a17b685f537c24e6c0d2 Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19954 Reviewed-by: Ali Alsuliman <[email protected]> Reviewed-by: Murtadha Hubail <[email protected]> Integration-Tests: Jenkins <[email protected]> Tested-by: Jenkins <[email protected]> --- M asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3ParallelDownloader.java 1 file changed, 36 insertions(+), 6 deletions(-) Approvals: Murtadha Hubail: Looks good to me, approved Ali Alsuliman: Looks good to me, but someone else must approve Jenkins: Verified; Verified Anon. E. Moose #1000171: diff --git a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3ParallelDownloader.java b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3ParallelDownloader.java index 4d27c5a..2acc12e 100644 --- a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3ParallelDownloader.java +++ b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3ParallelDownloader.java @@ -77,8 +77,7 @@ @Override public void downloadFiles(Collection<FileReference> toDownload) throws HyracksDataException { try { - List<CompletableFuture<CompletedFileDownload>> downloads = startDownloadingFiles(toDownload); - waitForFileDownloads(downloads); + downloadFilesAndWait(toDownload); } catch (IOException | ExecutionException | InterruptedException e) { throw HyracksDataException.create(e); } @@ -104,9 +103,10 @@ s3AsyncClient.close(); } - private List<CompletableFuture<CompletedFileDownload>> startDownloadingFiles(Collection<FileReference> toDownload) - throws IOException { + private void downloadFilesAndWait(Collection<FileReference> toDownload) + throws IOException, ExecutionException, InterruptedException { List<CompletableFuture<CompletedFileDownload>> downloads = new ArrayList<>(); + int maxPending = config.getRequestsMaxPendingHttpConnections(); for (FileReference fileReference : toDownload) { // multipart download profiler.objectGet(); @@ -126,13 +126,18 @@ FileDownload fileDownload = transferManager.downloadFile(builder.build()); downloads.add(fileDownload.completionFuture()); + if (maxPending > 0 && downloads.size() >= maxPending) { + waitForFileDownloads(downloads); + downloads.clear(); + } } - return downloads; + if (!downloads.isEmpty()) { + waitForFileDownloads(downloads); + } } private void waitForFileDownloads(List<CompletableFuture<CompletedFileDownload>> downloads) throws ExecutionException, InterruptedException { - for (CompletableFuture<CompletedFileDownload> download : downloads) { download.get(); } -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19954 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ifd1a30c6eed5f316a2d7a17b685f537c24e6c0d2 Gerrit-Change-Number: 19954 Gerrit-PatchSet: 3 Gerrit-Owner: Ali Alsuliman <[email protected]> Gerrit-Reviewer: Ali Alsuliman <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hussain Towaileb <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Michael Blow <[email protected]> Gerrit-Reviewer: Murtadha Hubail <[email protected]> Gerrit-Reviewer: Peeyush Gupta <[email protected]> Gerrit-MessageType: merged
