[
https://issues.apache.org/jira/browse/HADOOP-19102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825916#comment-17825916
]
ASF GitHub Bot commented on HADOOP-19102:
-----------------------------------------
saxenapranav commented on code in PR #6617:
URL: https://github.com/apache/hadoop/pull/6617#discussion_r1522581623
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java:
##########
@@ -71,22 +115,40 @@ public void
testMultipleServerCallsAreMadeWhenTheConfIsFalse()
private void testNumBackendCalls(boolean optimizeFooterRead)
throws Exception {
int fileIdx = 0;
- for (int i = 0; i <= 4; i++) {
- for (int j = 0; j <= 2; j++) {
- int fileSize = (int) Math.pow(2, i) * 256 * ONE_KB;
- int footerReadBufferSize = (int) Math.pow(2, j) * 256 * ONE_KB;
- final AzureBlobFileSystem fs = getFileSystem(
- optimizeFooterRead, fileSize);
- Path testFilePath = createPathAndFileWithContent(
- fs, fileIdx++, fileSize);
+ final List<Future> futureList = new ArrayList<>();
+ for (int fileSize : FILE_SIZES) {
+ final int fileId = fileIdx++;
+ Future future = executorService.submit(() -> {
+ try (AzureBlobFileSystem spiedFs = createSpiedFs(
+ getRawConfiguration())) {
+ Path testPath = createPathAndFileWithContent(
+ spiedFs, fileId, fileSize);
+ testNumBackendCalls(spiedFs, optimizeFooterRead, fileSize,
+ testPath);
+ } catch (Exception ex) {
+ throw new RuntimeException(ex);
+ }
+ });
+ futureList.add(future);
+ }
+ for (Future future : futureList) {
Review Comment:
Added two new APIs in FutureIO:
```
List<T> awaitFuture(Collection<Future<T>> collection, long timeout, TimeUnit
unit)
```
and
```
List<T> awaitFuture(Collection<Future<T>> collection)
```
> [ABFS]: FooterReadBufferSize should not be greater than readBufferSize
> ----------------------------------------------------------------------
>
> Key: HADOOP-19102
> URL: https://issues.apache.org/jira/browse/HADOOP-19102
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.4.0
> Reporter: Pranav Saxena
> Assignee: Pranav Saxena
> Priority: Major
> Labels: pull-request-available
>
> The method `optimisedRead` creates a buffer array of size `readBufferSize`.
> If footerReadBufferSize is greater than readBufferSize, abfs will attempt to
> read more data than the buffer array can hold, which causes an exception.
> Change: To avoid this, we will keep footerBufferSize =
> min(readBufferSizeConfig, footerBufferSizeConfig)
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
