[jira] [Updated] (HADOOP-13617) Swift client retrying original request is using expired token after re-authentication

2017-05-02 Thread Allen Wittenauer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-13617:
--
Fix Version/s: (was: 3.0.0-alpha1)

> Swift client retrying original request is using expired token after 
> re-authentication 
> --
>
> Key: HADOOP-13617
> URL: https://issues.apache.org/jira/browse/HADOOP-13617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
>Priority: Blocker
>  Labels: patch
> Attachments: 2016_09_13.stderrout.log, HADOOP-13617.patch
>
>
> library used: org.apache.hadoop:hadoop-openstack:2.6.0
> For long running Swift read operation (e.g., reading a large container), the 
> issued auth token has at most 30 minutes life span from Oracle Storage 
> Service. If the token expired in the middle of the read operation the 
> SwiftRestClient 
> (https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java#L1701)
>  re-authenticate and acquire a new auth token. However, in the retry request 
> the old, expired token is still used, causing the whole operation to fail.
> Because of this bug any meaningful(i.e., long-running) Swift operation is not 
> possible.
> Here is a summary of what happened with DEBUG logging turned on:
> ==
> 1. initial token acquired which will expire on 19:56:44(PDT; UTC-4):
> ---
> 2016-09-13 19:52:37 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk2dd9d639bbb992089dca008123c3046f',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@af28493,
> expires='2016-09-13T23:56:44Z'}
> 2. token expiration and re-authentication:
> --
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/
> X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:44 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable
> to respond to any of these challenges: {token=Token}
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 401
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1698 -
> Reauthenticating
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1079 - started
> authentication
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1228 -
> Authenticating with Authenticate as tenant 'Storage-paas132' user
> 'radha.sriniva...@oracle.com' with password of length 9
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - POST
> https://em2.storage.oraclecloud.com/auth/v2.0/tokens
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 200
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1149 - Catalog
> entry [swift: object-store];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1156 - Found
> swift catalog as swift => object-store
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1169 - Endpoint
> [US => https://em2.storage.oraclecloud.com/v1/Storage-paas132 / null];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d,
> expires='2016-09-14T00:26:45Z'}
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1216 -
> authenticated against https://em2.storage.oraclecloud.com/v1/Storage-paas132.
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - HEAD
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/
> X-Newest: true
> X-Auth-Token: AUTH_tk56bbb4d6fef57b7eeba7acae598f837c
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc

[jira] [Updated] (HADOOP-13617) Swift client retrying original request is using expired token after re-authentication

2016-10-09 Thread Yulei Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yulei Li updated HADOOP-13617:
--
   Labels: patch  (was: )
Fix Version/s: 3.0.0-alpha1
   Status: Patch Available  (was: Open)

> Swift client retrying original request is using expired token after 
> re-authentication 
> --
>
> Key: HADOOP-13617
> URL: https://issues.apache.org/jira/browse/HADOOP-13617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
>Priority: Blocker
>  Labels: patch
> Fix For: 3.0.0-alpha1
>
> Attachments: 2016_09_13.stderrout.log, HADOOP-13617.patch
>
>
> library used: org.apache.hadoop:hadoop-openstack:2.6.0
> For long running Swift read operation (e.g., reading a large container), the 
> issued auth token has at most 30 minutes life span from Oracle Storage 
> Service. If the token expired in the middle of the read operation the 
> SwiftRestClient 
> (https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java#L1701)
>  re-authenticate and acquire a new auth token. However, in the retry request 
> the old, expired token is still used, causing the whole operation to fail.
> Because of this bug any meaningful(i.e., long-running) Swift operation is not 
> possible.
> Here is a summary of what happened with DEBUG logging turned on:
> ==
> 1. initial token acquired which will expire on 19:56:44(PDT; UTC-4):
> ---
> 2016-09-13 19:52:37 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk2dd9d639bbb992089dca008123c3046f',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@af28493,
> expires='2016-09-13T23:56:44Z'}
> 2. token expiration and re-authentication:
> --
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/
> X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:44 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable
> to respond to any of these challenges: {token=Token}
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 401
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1698 -
> Reauthenticating
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1079 - started
> authentication
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1228 -
> Authenticating with Authenticate as tenant 'Storage-paas132' user
> 'radha.sriniva...@oracle.com' with password of length 9
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - POST
> https://em2.storage.oraclecloud.com/auth/v2.0/tokens
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 200
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1149 - Catalog
> entry [swift: object-store];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1156 - Found
> swift catalog as swift => object-store
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1169 - Endpoint
> [US => https://em2.storage.oraclecloud.com/v1/Storage-paas132 / null];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d,
> expires='2016-09-14T00:26:45Z'}
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1216 -
> authenticated against https://em2.storage.oraclecloud.com/v1/Storage-paas132.
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - HEAD
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/
> X-Newest: true
> X-Auth-Token: AUTH_tk56bbb4d6fef57b7eeba7acae598f837c
> User-Agent: Apache Hadoop Swift Client 2.6.0-c

[jira] [Updated] (HADOOP-13617) Swift client retrying original request is using expired token after re-authentication

2016-10-09 Thread Yulei Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yulei Li updated HADOOP-13617:
--
Attachment: HADOOP-13617.patch

> Swift client retrying original request is using expired token after 
> re-authentication 
> --
>
> Key: HADOOP-13617
> URL: https://issues.apache.org/jira/browse/HADOOP-13617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
>Priority: Blocker
> Attachments: 2016_09_13.stderrout.log, HADOOP-13617.patch
>
>
> library used: org.apache.hadoop:hadoop-openstack:2.6.0
> For long running Swift read operation (e.g., reading a large container), the 
> issued auth token has at most 30 minutes life span from Oracle Storage 
> Service. If the token expired in the middle of the read operation the 
> SwiftRestClient 
> (https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java#L1701)
>  re-authenticate and acquire a new auth token. However, in the retry request 
> the old, expired token is still used, causing the whole operation to fail.
> Because of this bug any meaningful(i.e., long-running) Swift operation is not 
> possible.
> Here is a summary of what happened with DEBUG logging turned on:
> ==
> 1. initial token acquired which will expire on 19:56:44(PDT; UTC-4):
> ---
> 2016-09-13 19:52:37 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk2dd9d639bbb992089dca008123c3046f',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@af28493,
> expires='2016-09-13T23:56:44Z'}
> 2. token expiration and re-authentication:
> --
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/
> X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:44 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable
> to respond to any of these challenges: {token=Token}
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 401
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1698 -
> Reauthenticating
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1079 - started
> authentication
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1228 -
> Authenticating with Authenticate as tenant 'Storage-paas132' user
> 'radha.sriniva...@oracle.com' with password of length 9
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - POST
> https://em2.storage.oraclecloud.com/auth/v2.0/tokens
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 200
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1149 - Catalog
> entry [swift: object-store];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1156 - Found
> swift catalog as swift => object-store
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1169 - Endpoint
> [US => https://em2.storage.oraclecloud.com/v1/Storage-paas132 / null];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d,
> expires='2016-09-14T00:26:45Z'}
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1216 -
> authenticated against https://em2.storage.oraclecloud.com/v1/Storage-paas132.
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - HEAD
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/
> X-Newest: true
> X-Auth-Token: AUTH_tk56bbb4d6fef57b7eeba7acae598f837c
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:45 DEB

[jira] [Updated] (HADOOP-13617) Swift client retrying original request is using expired token after re-authentication

2016-10-09 Thread Yulei Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yulei Li updated HADOOP-13617:
--
Attachment: (was: HADOOP-13617.patch)

> Swift client retrying original request is using expired token after 
> re-authentication 
> --
>
> Key: HADOOP-13617
> URL: https://issues.apache.org/jira/browse/HADOOP-13617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
>Priority: Blocker
> Attachments: 2016_09_13.stderrout.log, HADOOP-13617.patch
>
>
> library used: org.apache.hadoop:hadoop-openstack:2.6.0
> For long running Swift read operation (e.g., reading a large container), the 
> issued auth token has at most 30 minutes life span from Oracle Storage 
> Service. If the token expired in the middle of the read operation the 
> SwiftRestClient 
> (https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java#L1701)
>  re-authenticate and acquire a new auth token. However, in the retry request 
> the old, expired token is still used, causing the whole operation to fail.
> Because of this bug any meaningful(i.e., long-running) Swift operation is not 
> possible.
> Here is a summary of what happened with DEBUG logging turned on:
> ==
> 1. initial token acquired which will expire on 19:56:44(PDT; UTC-4):
> ---
> 2016-09-13 19:52:37 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk2dd9d639bbb992089dca008123c3046f',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@af28493,
> expires='2016-09-13T23:56:44Z'}
> 2. token expiration and re-authentication:
> --
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/
> X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:44 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable
> to respond to any of these challenges: {token=Token}
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 401
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1698 -
> Reauthenticating
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1079 - started
> authentication
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1228 -
> Authenticating with Authenticate as tenant 'Storage-paas132' user
> 'radha.sriniva...@oracle.com' with password of length 9
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - POST
> https://em2.storage.oraclecloud.com/auth/v2.0/tokens
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 200
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1149 - Catalog
> entry [swift: object-store];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1156 - Found
> swift catalog as swift => object-store
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1169 - Endpoint
> [US => https://em2.storage.oraclecloud.com/v1/Storage-paas132 / null];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d,
> expires='2016-09-14T00:26:45Z'}
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1216 -
> authenticated against https://em2.storage.oraclecloud.com/v1/Storage-paas132.
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - HEAD
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/
> X-Newest: true
> X-Auth-Token: AUTH_tk56bbb4d6fef57b7eeba7acae598f837c
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 1

[jira] [Updated] (HADOOP-13617) Swift client retrying original request is using expired token after re-authentication

2016-10-01 Thread Yulei Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yulei Li updated HADOOP-13617:
--
Attachment: HADOOP-13617.patch

To test the patch, one should set the fs.swift.token.expired in the 
auth-key.xml to be greater than the expiration time of your environment, and if 
the token expiration time is greater than 900s, one should change the value of 
surefire.fork.timeout in hadoop-project/pom.xml to be greater than the 
expiration time, or the test will failed.

> Swift client retrying original request is using expired token after 
> re-authentication 
> --
>
> Key: HADOOP-13617
> URL: https://issues.apache.org/jira/browse/HADOOP-13617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
>Priority: Blocker
> Attachments: 2016_09_13.stderrout.log, HADOOP-13617.patch
>
>
> library used: org.apache.hadoop:hadoop-openstack:2.6.0
> For long running Swift read operation (e.g., reading a large container), the 
> issued auth token has at most 30 minutes life span from Oracle Storage 
> Service. If the token expired in the middle of the read operation the 
> SwiftRestClient 
> (https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java#L1701)
>  re-authenticate and acquire a new auth token. However, in the retry request 
> the old, expired token is still used, causing the whole operation to fail.
> Because of this bug any meaningful(i.e., long-running) Swift operation is not 
> possible.
> Here is a summary of what happened with DEBUG logging turned on:
> ==
> 1. initial token acquired which will expire on 19:56:44(PDT; UTC-4):
> ---
> 2016-09-13 19:52:37 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk2dd9d639bbb992089dca008123c3046f',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@af28493,
> expires='2016-09-13T23:56:44Z'}
> 2. token expiration and re-authentication:
> --
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/
> X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:44 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable
> to respond to any of these challenges: {token=Token}
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 401
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1698 -
> Reauthenticating
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1079 - started
> authentication
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1228 -
> Authenticating with Authenticate as tenant 'Storage-paas132' user
> 'radha.sriniva...@oracle.com' with password of length 9
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - POST
> https://em2.storage.oraclecloud.com/auth/v2.0/tokens
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 200
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1149 - Catalog
> entry [swift: object-store];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1156 - Found
> swift catalog as swift => object-store
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1169 - Endpoint
> [US => https://em2.storage.oraclecloud.com/v1/Storage-paas132 / null];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d,
> expires='2016-09-14T00:26:45Z'}
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1216 -
> authenticated against https://em2.storage.oraclecloud.com/v1/Storage-paas132.
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:

[jira] [Updated] (HADOOP-13617) Swift client retrying original request is using expired token after re-authentication

2016-09-21 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13617:

Assignee: Yulei Li

> Swift client retrying original request is using expired token after 
> re-authentication 
> --
>
> Key: HADOOP-13617
> URL: https://issues.apache.org/jira/browse/HADOOP-13617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Assignee: Yulei Li
>Priority: Blocker
> Attachments: 2016_09_13.stderrout.log
>
>
> library used: org.apache.hadoop:hadoop-openstack:2.6.0
> For long running Swift read operation (e.g., reading a large container), the 
> issued auth token has at most 30 minutes life span from Oracle Storage 
> Service. If the token expired in the middle of the read operation the 
> SwiftRestClient 
> (https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java#L1701)
>  re-authenticate and acquire a new auth token. However, in the retry request 
> the old, expired token is still used, causing the whole operation to fail.
> Because of this bug any meaningful(i.e., long-running) Swift operation is not 
> possible.
> Here is a summary of what happened with DEBUG logging turned on:
> ==
> 1. initial token acquired which will expire on 19:56:44(PDT; UTC-4):
> ---
> 2016-09-13 19:52:37 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk2dd9d639bbb992089dca008123c3046f',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@af28493,
> expires='2016-09-13T23:56:44Z'}
> 2. token expiration and re-authentication:
> --
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/
> X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:44 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable
> to respond to any of these challenges: {token=Token}
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 401
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1698 -
> Reauthenticating
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1079 - started
> authentication
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1228 -
> Authenticating with Authenticate as tenant 'Storage-paas132' user
> 'radha.sriniva...@oracle.com' with password of length 9
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - POST
> https://em2.storage.oraclecloud.com/auth/v2.0/tokens
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 200
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1149 - Catalog
> entry [swift: object-store];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1156 - Found
> swift catalog as swift => object-store
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1169 - Endpoint
> [US => https://em2.storage.oraclecloud.com/v1/Storage-paas132 / null];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d,
> expires='2016-09-14T00:26:45Z'}
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1216 -
> authenticated against https://em2.storage.oraclecloud.com/v1/Storage-paas132.
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - HEAD
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/
> X-Newest: true
> X-Auth-Token: AUTH_tk56bbb4d6fef57b7eeba7acae598f837c
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1]

[jira] [Updated] (HADOOP-13617) Swift client retrying original request is using expired token after re-authentication

2016-09-16 Thread Steve Yang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Yang updated HADOOP-13617:

Priority: Blocker  (was: Major)

> Swift client retrying original request is using expired token after 
> re-authentication 
> --
>
> Key: HADOOP-13617
> URL: https://issues.apache.org/jira/browse/HADOOP-13617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Priority: Blocker
> Attachments: 2016_09_13.stderrout.log
>
>
> library used: org.apache.hadoop:hadoop-openstack:2.6.0
> For long running Swift read operation (e.g., reading a large container), the 
> issued auth token has at most 30 minutes life span from Oracle Storage 
> Service. If the token expired in the middle of the read operation the 
> SwiftRestClient 
> (https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java#L1701)
>  re-authenticate and acquire a new auth token. However, in the retry request 
> the old, expired token is still used, causing the whole operation to fail.
> Because of this bug any meaningful(i.e., long-running) Swift operation is not 
> possible.
> Here is a summary of what happened with DEBUG logging turned on:
> ==
> 1. initial token acquired which will expire on 19:56:44(PDT; UTC-4):
> ---
> 2016-09-13 19:52:37 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk2dd9d639bbb992089dca008123c3046f',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@af28493,
> expires='2016-09-13T23:56:44Z'}
> 2. token expiration and re-authentication:
> --
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/
> X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:44 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable
> to respond to any of these challenges: {token=Token}
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 401
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1698 -
> Reauthenticating
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1079 - started
> authentication
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1228 -
> Authenticating with Authenticate as tenant 'Storage-paas132' user
> 'radha.sriniva...@oracle.com' with password of length 9
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - POST
> https://em2.storage.oraclecloud.com/auth/v2.0/tokens
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 200
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1149 - Catalog
> entry [swift: object-store];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1156 - Found
> swift catalog as swift => object-store
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1169 - Endpoint
> [US => https://em2.storage.oraclecloud.com/v1/Storage-paas132 / null];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d,
> expires='2016-09-14T00:26:45Z'}
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1216 -
> authenticated against https://em2.storage.oraclecloud.com/v1/Storage-paas132.
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - HEAD
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/
> X-Newest: true
> X-Auth-Token: AUTH_tk56bbb4d6fef57b7eeba7acae598f837c
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Sta

[jira] [Updated] (HADOOP-13617) Swift client retrying original request is using expired token after re-authentication

2016-09-16 Thread Steve Yang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Yang updated HADOOP-13617:

Attachment: 2016_09_13.stderrout.log

complete log file showing Swift Rest client's interaction with Oracle Storage 
Service server (i.e., Swift server)

> Swift client retrying original request is using expired token after 
> re-authentication 
> --
>
> Key: HADOOP-13617
> URL: https://issues.apache.org/jira/browse/HADOOP-13617
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
> Attachments: 2016_09_13.stderrout.log
>
>
> library used: org.apache.hadoop:hadoop-openstack:2.6.0
> For long running Swift read operation (e.g., reading a large container), the 
> issued auth token has at most 30 minutes life span from Oracle Storage 
> Service. If the token expired in the middle of the read operation the 
> SwiftRestClient 
> (https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java#L1701)
>  re-authenticate and acquire a new auth token. However, in the retry request 
> the old, expired token is still used, causing the whole operation to fail.
> Because of this bug any meaningful(i.e., long-running) Swift operation is not 
> possible.
> Here is a summary of what happened with DEBUG logging turned on:
> ==
> 1. initial token acquired which will expire on 19:56:44(PDT; UTC-4):
> ---
> 2016-09-13 19:52:37 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk2dd9d639bbb992089dca008123c3046f',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@af28493,
> expires='2016-09-13T23:56:44Z'}
> 2. token expiration and re-authentication:
> --
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/&format=json&delimiter=/
> X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:44 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable
> to respond to any of these challenges: {token=Token}
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 401
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1698 -
> Reauthenticating
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1079 - started
> authentication
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1228 -
> Authenticating with Authenticate as tenant 'Storage-paas132' user
> 'radha.sriniva...@oracle.com' with password of length 9
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - POST
> https://em2.storage.oraclecloud.com/auth/v2.0/tokens
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 200
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1149 - Catalog
> entry [swift: object-store];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1156 - Found
> swift catalog as swift => object-store
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1169 - Endpoint
> [US => https://em2.storage.oraclecloud.com/v1/Storage-paas132 / null];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d,
> expires='2016-09-14T00:26:45Z'}
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1216 -
> authenticated against https://em2.storage.oraclecloud.com/v1/Storage-paas132.
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - HEAD
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/
> X-Newest: true
> X-Auth-Token: AUTH_tk56bbb4d6fef57b7eeba7acae598f837c
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f0