Steve Yang created HADOOP-13617: ----------------------------------- Summary: Swift client retrying original request is using expired token after re-authentication Key: HADOOP-13617 URL: https://issues.apache.org/jira/browse/HADOOP-13617 Project: Hadoop Common Issue Type: Bug Components: fs/swift Affects Versions: 2.6.0 Environment: Linux EL6 Reporter: Steve Yang
library used: org.apache.hadoop:hadoop-openstack:2.6.0 For long running Swift read operation (e.g., reading a large container), the issued auth token has at most 30 minutes life span from Oracle Storage Service. If the token expired in the middle of the read operation the SwiftRestClient (https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java#L1701) re-authenticate and acquire a new auth token. However, in the retry request the old, expired token is still used, causing the whole operation to fail. Because of this bug any meaningful(i.e., long-running) Swift operation is not possible. Here is a summary of what happened with DEBUG logging turned on: ========================================================== 1. initial token acquired which will expire on 19:56:44(PDT; UTC-4): ------------------------------------------------------------------- 2016-09-13 19:52:37 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth: endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132; objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null; token=AccessToken{id='AUTH_tk2dd9d639bbb992089dca008123c3046f', tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@af28493, expires='2016-09-13T23:56:44Z'} 2. token expiration and re-authentication: ------------------------------------------ 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/ X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum 298b68dc3b308983f04cb37e8416f13 . 2016-09-13 19:56:44 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable to respond to any of these challenges: {token=Token} 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status code = 401 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1698 - Reauthenticating 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1079 - started authentication 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1228 - Authenticating with Authenticate as tenant 'Storage-paas132' user 'radha.sriniva...@oracle.com' with password of length 9 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - POST https://em2.storage.oraclecloud.com/auth/v2.0/tokens User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum 298b68dc3b308983f04cb37e8416f13 . 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status code = 200 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1149 - Catalog entry [swift: object-store]; 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1156 - Found swift catalog as swift => object-store 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1169 - Endpoint [US => https://em2.storage.oraclecloud.com/v1/Storage-paas132 / null]; 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth: endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132; objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null; token=AccessToken{id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c', tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d, expires='2016-09-14T00:26:45Z'} 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1216 - authenticated against https://em2.storage.oraclecloud.com/v1/Storage-paas132. 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - HEAD https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/ X-Newest: true X-Auth-Token: AUTH_tk56bbb4d6fef57b7eeba7acae598f837c User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum 298b68dc3b308983f04cb37e8416f13 . 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status code = 204 3. retrying the original request but the auth token used is the old one: ----------------------------------------------------------------------- 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1703 - Retrying original request 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/ X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum 298b68dc3b308983f04cb37e8416f13 Host: em2.storage.oraclecloud.com . 2016-09-13 19:56:46 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable to respond to any of these challenges: {token=Token} 2016-09-13 19:56:46 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status code = 401 2016-09-13 19:56:46 DEBUG [pool-3-thread-1] SwiftRestClient:1445 - Method GET on https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/ failed, status code: 401, status line: HTTP/1.1 401 Unauthorized 2016-09-13 19:56:46 ERROR [pool-3-thread-1] SnapshotJob:244 - Error reading snapshot Authentication Failure: Operation not authorized- current access token =AccessToken{id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c', tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d, expires='2016-09-14T00:26:45Z'} GET https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/ => 401 : Token not found in cache at org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient .java:1482) at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1 403) at org.apache.hadoop.fs.swift.http.SwiftRestClient.findObjects(SwiftRestClient.ja va:881) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org