[
https://issues.apache.org/jira/browse/HADOOP-15478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481157#comment-16481157
]
Thomas Marquardt commented on HADOOP-15478:
-------------------------------------------
I've attached HADOOP-15478.001.patch with the fix and new test cases. The new
testBlockBlobHFlush() and testBlockBlobHSync() tests fail without the fix, and
pass with the fix. The new tests ensure correct behavior of flush, hflush,
hsync, and close for the output streams used by the WASB driver.
WASB test results against my Azure storage account are below:
*$ mvn -T 1C -Dparallel-tests clean verify*
Tests run: 241, Failures: 0, Errors: 0, Skipped: 11
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
Tests run: 444, Failures: 0, Errors: 0, Skipped: 55
Tests run: 126, Failures: 3, Errors: 0, Skipped: 10
Note 3 tests are failing in trunk. These are failing without my change, and I
will look into them.
*[ERROR]
ITestNativeAzureFileSystemLive>NativeAzureFileSystemBaseTest.testFolderLastModifiedTime*:
651->NativeAzureFileSystemBaseTest.testModifiedTime:1492->Assert.assertTrue:41->Assert.fail:88
Modification time Fri May 18 20:21:56 UTC 2018 is not close to now: Fri May 18
20:21:40 UTC 2018
*[ERROR]
ITestNativeAzureFileSystemLive>NativeAzureFileSystemBaseTest.testModifiedTimeForFile*:
626->NativeAzureFileSystemBaseTest.testModifiedTime:1492->Assert.assertTrue:41->Assert.fail:88
Modification time Fri May 18 20:23:31 UTC 2018 is not close to now: Fri May 18
20:23:15 UTC 2018
*[ERROR]
ITestNativeAzureFileSystemLive>NativeAzureFileSystemBaseTest.testModifiedTimeForFolder*:
633->NativeAzureFileSystemBaseTest.testModifiedTime:1492->Assert.assertTrue:41->Assert.fail:88
Modification time Fri May 18 20:21:49 UTC 2018 is not close to now: Fri May 18
20:21:33 UTC 2018
> WASB: hflush() and hsync() regression
> -------------------------------------
>
> Key: HADOOP-15478
> URL: https://issues.apache.org/jira/browse/HADOOP-15478
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Affects Versions: 2.9.0, 3.0.2
> Reporter: Thomas Marquardt
> Assignee: Thomas Marquardt
> Priority: Major
> Attachments: HADOOP-15478.001.patch
>
>
> HADOOP-14520 introduced a regression in hflush() and hsync(). Previously,
> for the default case where users upload data as block blobs, these were
> no-ops. Unfortunately, HADOOP-14520 accidentally implemented hflush() and
> hsync() by default, so any data buffered in the stream is immediately
> uploaded to storage. This new behavior is undesirable, because block blobs
> have a limit of 50,000 blocks. Spark users are now seeing failures due to
> exceeding the block limit, since Spark frequently invokes hflush().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
