[
https://issues.apache.org/jira/browse/HADOOP-19102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825186#comment-17825186
]
ASF GitHub Bot commented on HADOOP-19102:
-----------------------------------------
saxenapranav commented on code in PR #6617:
URL: https://github.com/apache/hadoop/pull/6617#discussion_r1519264819
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java:
##########
@@ -167,24 +214,55 @@ public void testSeekToEndAndReadWithConfFalse() throws
Exception {
private void testSeekAndReadWithConf(boolean optimizeFooterRead,
SeekTo seekTo) throws Exception {
+ int fileIdx = 0;
+ List<Future> futureList = new ArrayList<>();
+ for (int j = 0; j <= 4; j++) {
+ final int fileSize = (int) Math.pow(2, j) * SIZE_256_KB;
+ final int fileId = fileIdx++;
+ futureList.add(executorService.submit(() -> {
+ try {
+ try (AzureBlobFileSystem spiedFs = createSpiedFs(
+ getRawConfiguration())) {
+ String fileName = methodName.getMethodName() + fileId;
+ byte[] fileContent = getRandomBytesArray(fileSize);
+ Path testFilePath = createFileWithContent(spiedFs, fileName,
+ fileContent);
+ for (int i = 0; i <= 4; i++) {
Review Comment:
ReadBufferSize has default equal to 4 MB. Will keep this in range. Though
have removed 2 MB from readBufferSize and fileSize. Following are the ranges:
fileSize: 256 KB, 512 KB, 1 MB, 4 MB
readBufferSize: 256 KB, 512 KB, 1 MB, 4 MB
footerReadBufferSize: 256 KB, 512 KB, 1 MB,
> [ABFS]: FooterReadBufferSize should not be greater than readBufferSize
> ----------------------------------------------------------------------
>
> Key: HADOOP-19102
> URL: https://issues.apache.org/jira/browse/HADOOP-19102
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Reporter: Pranav Saxena
> Assignee: Pranav Saxena
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.5.0
>
>
> The method `optimisedRead` creates a buffer array of size `readBufferSize`.
> If footerReadBufferSize is greater than readBufferSize, abfs will attempt to
> read more data than the buffer array can hold, which causes an exception.
> Change: To avoid this, we will keep footerBufferSize =
> min(readBufferSizeConfig, footerBufferSizeConfig)
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
