[
https://issues.apache.org/jira/browse/HDFS-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jay Hankinson updated HDFS-13518:
---------------------------------
Description:
If hdfsFileClose() is called on a file opened for write/append directly after
hdfsHSync(), without further data being written to the file, the file doesn't
close and the lease isn't released. In the ADL
output trace, with hdfsHSync() immediately before hdfsFileClose() I see:
18/04/26 23:31:11 DEBUG store.HttpTransport:
HTTPRequest,Succeeded,cReqId:607c6051-2ef3-4e57-9b20-caea70f22a31.0,lat:43,err:,Reqlen:14,Resplen:0,token_ns:3700,sReqId:40b2a8df-c9b5-4eff-b84e-c1346f4fdb88,path:/tmp/testfile.txt,qp:op=APPEND&append=true&syncFlag=METADATA&filesessionid=76b19d5b-f3ad-4400-a742-2f7996450859&leaseid=76b19d5b-f3ad-4400-a742-2f7996450859&offset=28&api-version=2016-11-01
but nothing else. I also get an error if I try to re-open the file:
18/04/26 23:37:28 DEBUG store.HttpTransport:
HTTPRequest,Failed,cReqId:f216b9d6-3e35-4166-92cf-a7c4e2389dcd.0,lat:9,err:HTTP400(IllegalArgumentException),Reqlen:0,Resplen:357,token_ns:13001,sReqId:4b9e1e11-3f61-46a6-b461-3e5b180266c5,path:/tmp/testfile.txt,qp:op=APPEND&append=true&syncFlag=DATA&filesessionid=3197c37c-356d-491a-a902-aa19b6d8a374&leaseid=3197c37c-356d-491a-a902-aa19b6d8a374&api-version=2016-11-01
hdfsOpenFile(/tmp/testfile.txt):
FileSystem#append((Lorg/apache/hadoop/fs/Path;)Lorg/apache/hadoop/fs/FSDataOutputStream;)
error:
java.io.IOException: APPEND failed with error 0x83090a16 (Failed to perform
the requested operation because the file is currently open in write mode by
another user or process.).
[4b9e1e11-3f61-46a6-b461-3e5b180266c5][2018-04-26T16:37:28.6549201-07:00]
at
com.microsoft.azure.datalake.store.ADLStoreClient.getRemoteException(ADLStoreClient.java:1142)
at
com.microsoft.azure.datalake.store.ADLStoreClient.getExceptionFromResponse(ADLStoreClient.java:1106)
at
com.microsoft.azure.datalake.store.ADLStoreClient.getAppendStream(ADLStoreClient.java:294)
at org.apache.hadoop.fs.adl.AdlFileSystem.append(AdlFileSystem.java:398)
at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1187)
Appending to the file is then prevented for the next 8 or so minutes until the
lease expires. If hdfsWrite()/flush() is called after hdfsHSync() and before
hdfsFileClose() I see the APPEND() with syncFlag=CLOSE as expected.
18/04/26 23:37:53 DEBUG store.HttpTransport:
HTTPRequest,Succeeded,cReqId:ecd19d36-b535-4dd5-8c49-1588a2a60947.0,lat:124,err:,Reqlen:14,Resplen:0,token_ns:4700,sReqId:4a6a2c94-ab31-47b2-aae8-1e4250974d27,path:/tmp/testfile.txt,qp:op=APPEND&append=true&syncFlag=METADATA&filesessionid=21adf26e-3208-40fc-803a-1c18e1914090&leaseid=21adf26e-3208-40fc-803a-1c18e1914090&offset=0&api-version=2016-11-01
18/04/26 23:37:53 DEBUG store.HttpTransport:
HTTPRequest,Succeeded,cReqId:08708059-6396-4af9-a4da-d7d60c44041b.0,lat:43,err:,Reqlen:14,Resplen:0,token_ns:3900,sReqId:a4b5f419-f87f-4bed-99ed-8ad1d24c9ab7,path:/tmp/testfile.txt,qp:op=APPEND&append=true&syncFlag=CLOSE&filesessionid=21adf26e-3208-40fc-803a-1c18e1914090&leaseid=21adf26e-3208-40fc-803a-1c18e1914090&offset=14&api-version=2016-11-01
libhdfs repro case attached.
was:
If hdfsFileClose() is called on a file opened for write/append directly after
hdfsHSync(), without further data being written to the file, the file doesn't
close and the lease isn't released. In the ADL
output trace, with hdfsHSync() immediately before hdfsFileClose() I see:
18/04/26 23:31:11 DEBUG store.HttpTransport:
HTTPRequest,Succeeded,cReqId:607c6051-2ef3-4e57-9b20-caea70f22a31.0,lat:43,err:,Reqlen:14,Resplen:0,token_ns:3700,sReqId:40b2a8df-c9b5-4eff-b84e-c1346f4fdb88,path:/tmp/testfile.txt,qp:op=APPEND&append=true&syncFlag=METADATA&filesessionid=76b19d5b-f3ad-4400-a742-2f7996450859&leaseid=76b19d5b-f3ad-4400-a742-2f7996450859&offset=28&api-version=2016-11-01
but nothing else. I also get an error if I try to re-open the file:
18/04/26 23:37:28 DEBUG store.HttpTransport:
HTTPRequest,Failed,cReqId:f216b9d6-3e35-4166-92cf-a7c4e2389dcd.0,lat:9,err:HTTP400(IllegalArgumentException),Reqlen:0,Resplen:357,token_ns:13001,sReqId:4b9e1e11-3f61-46a6-b461-3e5b180266c5,path:/tmp/testfile.txt,qp:op=APPEND&append=true&syncFlag=DATA&filesessionid=3197c37c-356d-491a-a902-aa19b6d8a374&leaseid=3197c37c-356d-491a-a902-aa19b6d8a374&api-version=2016-11-01
hdfsOpenFile(/tmp/testfile.txt):
FileSystem#append((Lorg/apache/hadoop/fs/Path;)Lorg/apache/hadoop/fs/FSDataOutputStream;)
error:
java.io.IOException: APPEND failed with error 0x83090a16 (Failed to perform the
requested operation because the file is currently open in write mode by another
user or process.).
[4b9e1e11-3f61-46a6-b461-3e5b180266c5][2018-04-26T16:37:28.6549201-07:00]
at
com.microsoft.azure.datalake.store.ADLStoreClient.getRemoteException(ADLStoreClient.java:1142)
at
com.microsoft.azure.datalake.store.ADLStoreClient.getExceptionFromResponse(ADLStoreClient.java:1106)
at
com.microsoft.azure.datalake.store.ADLStoreClient.getAppendStream(ADLStoreClient.java:294)
at org.apache.hadoop.fs.adl.AdlFileSystem.append(AdlFileSystem.java:398)
at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1187)
If hdfsWrite()/flush() is called after hdfsHSync() and before hdfsFileClose() I
see the APPEND() with syncFlag=CLOSE as expected.
18/04/26 23:37:53 DEBUG store.HttpTransport:
HTTPRequest,Succeeded,cReqId:ecd19d36-b535-4dd5-8c49-1588a2a60947.0,lat:124,err:,Reqlen:14,Resplen:0,token_ns:4700,sReqId:4a6a2c94-ab31-47b2-aae8-1e4250974d27,path:/tmp/testfile.txt,qp:op=APPEND&append=true&syncFlag=METADATA&filesessionid=21adf26e-3208-40fc-803a-1c18e1914090&leaseid=21adf26e-3208-40fc-803a-1c18e1914090&offset=0&api-version=2016-11-01
18/04/26 23:37:53 DEBUG store.HttpTransport:
HTTPRequest,Succeeded,cReqId:08708059-6396-4af9-a4da-d7d60c44041b.0,lat:43,err:,Reqlen:14,Resplen:0,token_ns:3900,sReqId:a4b5f419-f87f-4bed-99ed-8ad1d24c9ab7,path:/tmp/testfile.txt,qp:op=APPEND&append=true&syncFlag=CLOSE&filesessionid=21adf26e-3208-40fc-803a-1c18e1914090&leaseid=21adf26e-3208-40fc-803a-1c18e1914090&offset=14&api-version=2016-11-01
Appending to the file is then prevented for the next 8 or so minutes until the
lease expires. libhdfs repro case attached.
> hdfsHSync()/hdfsFileClose() doesn't release lease with libhdfs on Azure ADL
> ---------------------------------------------------------------------------
>
> Key: HDFS-13518
> URL: https://issues.apache.org/jira/browse/HDFS-13518
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
> Affects Versions: 2.7.3
> Environment: HDInsight in Azure:
> actian@hn0-vector:~$ hadoop version
> Hadoop 2.7.3.2.6.2.25-1
> Subversion [email protected]:hortonworks/hadoop.git -r
> 1ceeb58bb3bb5904df0cbb7983389bcaf2ffd0b6
> Compiled by jenkins on 2017-11-29T15:28Z
> Compiled with protoc 2.5.0
> From source with checksum 90b73c4c185645c1f47b61f942230
> This command was run using
> /usr/hdp/2.6.2.25-1/hadoop/hadoop-common-2.7.3.2.6.2.25-1.jar
> actian@hn0-vector:~$ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=16.04
> DISTRIB_CODENAME=xenial
> DISTRIB_DESCRIPTION="Ubuntu 16.04.4 LTS"
> Reporter: Jay Hankinson
> Priority: Major
> Attachments: hdfssimple.c
>
>
> If hdfsFileClose() is called on a file opened for write/append directly after
> hdfsHSync(), without further data being written to the file, the file doesn't
> close and the lease isn't released. In the ADL
> output trace, with hdfsHSync() immediately before hdfsFileClose() I see:
>
> 18/04/26 23:31:11 DEBUG store.HttpTransport:
> HTTPRequest,Succeeded,cReqId:607c6051-2ef3-4e57-9b20-caea70f22a31.0,lat:43,err:,Reqlen:14,Resplen:0,token_ns:3700,sReqId:40b2a8df-c9b5-4eff-b84e-c1346f4fdb88,path:/tmp/testfile.txt,qp:op=APPEND&append=true&syncFlag=METADATA&filesessionid=76b19d5b-f3ad-4400-a742-2f7996450859&leaseid=76b19d5b-f3ad-4400-a742-2f7996450859&offset=28&api-version=2016-11-01
>
> but nothing else. I also get an error if I try to re-open the file:
>
> 18/04/26 23:37:28 DEBUG store.HttpTransport:
> HTTPRequest,Failed,cReqId:f216b9d6-3e35-4166-92cf-a7c4e2389dcd.0,lat:9,err:HTTP400(IllegalArgumentException),Reqlen:0,Resplen:357,token_ns:13001,sReqId:4b9e1e11-3f61-46a6-b461-3e5b180266c5,path:/tmp/testfile.txt,qp:op=APPEND&append=true&syncFlag=DATA&filesessionid=3197c37c-356d-491a-a902-aa19b6d8a374&leaseid=3197c37c-356d-491a-a902-aa19b6d8a374&api-version=2016-11-01
> hdfsOpenFile(/tmp/testfile.txt):
> FileSystem#append((Lorg/apache/hadoop/fs/Path;)Lorg/apache/hadoop/fs/FSDataOutputStream;)
> error:
> java.io.IOException: APPEND failed with error 0x83090a16 (Failed to perform
> the requested operation because the file is currently open in write mode by
> another user or process.).
> [4b9e1e11-3f61-46a6-b461-3e5b180266c5][2018-04-26T16:37:28.6549201-07:00]
> at
> com.microsoft.azure.datalake.store.ADLStoreClient.getRemoteException(ADLStoreClient.java:1142)
> at
> com.microsoft.azure.datalake.store.ADLStoreClient.getExceptionFromResponse(ADLStoreClient.java:1106)
> at
> com.microsoft.azure.datalake.store.ADLStoreClient.getAppendStream(ADLStoreClient.java:294)
> at org.apache.hadoop.fs.adl.AdlFileSystem.append(AdlFileSystem.java:398)
> at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1187)
>
> Appending to the file is then prevented for the next 8 or so minutes until
> the lease expires. If hdfsWrite()/flush() is called after hdfsHSync() and
> before hdfsFileClose() I see the APPEND() with syncFlag=CLOSE as expected.
>
> 18/04/26 23:37:53 DEBUG store.HttpTransport:
> HTTPRequest,Succeeded,cReqId:ecd19d36-b535-4dd5-8c49-1588a2a60947.0,lat:124,err:,Reqlen:14,Resplen:0,token_ns:4700,sReqId:4a6a2c94-ab31-47b2-aae8-1e4250974d27,path:/tmp/testfile.txt,qp:op=APPEND&append=true&syncFlag=METADATA&filesessionid=21adf26e-3208-40fc-803a-1c18e1914090&leaseid=21adf26e-3208-40fc-803a-1c18e1914090&offset=0&api-version=2016-11-01
> 18/04/26 23:37:53 DEBUG store.HttpTransport:
> HTTPRequest,Succeeded,cReqId:08708059-6396-4af9-a4da-d7d60c44041b.0,lat:43,err:,Reqlen:14,Resplen:0,token_ns:3900,sReqId:a4b5f419-f87f-4bed-99ed-8ad1d24c9ab7,path:/tmp/testfile.txt,qp:op=APPEND&append=true&syncFlag=CLOSE&filesessionid=21adf26e-3208-40fc-803a-1c18e1914090&leaseid=21adf26e-3208-40fc-803a-1c18e1914090&offset=14&api-version=2016-11-01
>
> libhdfs repro case attached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
