I ran put blob operation of 1 KB and received 200 OK response but when I queried for that blob after some time, it gave me 404 not found error. It looks like the IOPS-4 transfer was not at all successful at first hand even though it reported 200 OK.
This issue is not seen for every run but happens very infrequently at times. What could be the problem here? jclouds debug logs are pasted below. *2017-03-14@07:41:54.372 D [validate-34 ] pCommandExecutorService:logDebug - Sending request -341731354: PUT https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com/IOPS-4 <https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com/IOPS-4> HTTP/1.1* 2017-03-14@07:41:54.374 D [validate-34 ] wire:logDebug - >> "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" 2017-03-14@07:41:54.374 D [validate-34 ] headers:logDebug - >> PUT https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com/IOPS-4 HTTP/1.1 2017-03-14@07:41:54.374 D [validate-34 ] headers:logDebug - >> Host: mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com 2017-03-14@07:41:54.374 D [validate-34 ] headers:logDebug - >> x-amz-meta-ics.stack-description: ICStore.1(BlobStoreConnection.1) 2017-03-14@07:41:54.375 D [validate-34 ] headers:logDebug - >> x-amz-meta-ics.meta-digest: 8bd279c4f237299a539cef43a02f3cfc 2017-03-14@07:41:54.375 D [validate-34 ] headers:logDebug - >> x-amz-meta-ics.meta-version: 1 2017-03-14@07:41:54.375 D [validate-34 ] headers:logDebug - >> Date: Tue, 14 Mar 2017 11:41:41 GMT 2017-03-14@07:41:54.375 D [validate-34 ] headers:logDebug - >> Authorization: AWS AKIAJFXGJAH3NFZLJMWQ:DO1PzrgSpA5aK+5ogXIfocl+qkY= 2017-03-14@07:41:54.375 D [validate-34 ] headers:logDebug - >> Content-Type: application/unknown 2017-03-14@07:41:54.375 D [validate-34 ] headers:logDebug - >> Content-Length: 1024 2017-03-14@07:41:54.464 D [validate-34 ] pCommandExecutorService:logDebug - Receiving response -2056154209: HTTP/1.1 307 Temporary Redirect 2017-03-14@07:41:54.464 D [validate-34 ] headers:logDebug - << HTTP/1.1 307 Temporary Redirect 2017-03-14@07:41:54.464 D [validate-34 ] headers:logDebug - << Transfer-Encoding: chunked 2017-03-14@07:41:54.464 D [validate-34 ] headers:logDebug - << Server: AmazonS3 2017-03-14@07:41:54.465 D [validate-34 ] headers:logDebug - << Connection: close 2017-03-14@07:41:54.465 D [validate-34 ] headers:logDebug - << x-amz-request-id: 9A843CB77D2F5B90 2017-03-14@07:41:54.465 D [validate-34 ] headers:logDebug - << x-amz-id-2: JCga+UZjgKmtvTTEzOXSaeOqRO3Spf3K2tcxBstqXWIkVVDKUMXSRH4cUbAiY+3H 2017-03-14@07:41:54.465 D [validate-34 ] headers:logDebug - << Date: Tue, 14 Mar 2017 11:41:52 GMT 2017-03-14@07:41:54.465 D [validate-34 ] headers:logDebug - << x-amz-bucket-region: ap-northeast-1 2017-03-14@07:41:54.466 D [validate-34 ] headers:logDebug - << Location: https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4 2017-03-14@07:41:54.466 D [validate-34 ] headers:logDebug - << Content-Type: application/xml 2017-03-14@07:41:54.467 D [validate-34 ] wire:logDebug - << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 2017-03-14@07:41:54.468 D [validate-34 ] wire:logDebug - << "<Error><Code>TemporaryRedirect</Code><Message>Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.</Message><Bucket>mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918</Bucket><Endpoint> mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com </Endpoint><RequestId>9A843CB77D2F5B90</RequestId><HostId>JCga+UZjgKmtvTTEzOXSaeOqRO3Spf3K2tcxBstqXWIkVVDKUMXSRH4cUbAiY+3H</HostId></Error>" *2017-03-14@07:41:54.470 D [validate-34 ] pCommandExecutorService:logDebug - Sending request -593084728: PUT https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4 <https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4> HTTP/1.1* 2017-03-14@07:41:54.472 D [validate-34 ] wire:logDebug - >> "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" 2017-03-14@07:41:54.473 D [validate-34 ] headers:logDebug - >> PUT https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4 HTTP/1.1 2017-03-14@07:41:54.473 D [validate-34 ] headers:logDebug - >> x-amz-meta-ics.stack-description: ICStore.1(BlobStoreConnection.1) 2017-03-14@07:41:54.473 D [validate-34 ] headers:logDebug - >> x-amz-meta-ics.meta-digest: 8bd279c4f237299a539cef43a02f3cfc 2017-03-14@07:41:54.473 D [validate-34 ] headers:logDebug - >> x-amz-meta-ics.meta-version: 1 2017-03-14@07:41:54.474 D [validate-34 ] headers:logDebug - >> Host: mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com 2017-03-14@07:41:54.474 D [validate-34 ] headers:logDebug - >> Date: Tue, 14 Mar 2017 11:41:41 GMT 2017-03-14@07:41:54.474 D [validate-34 ] headers:logDebug - >> Authorization: AWS AKIAJFXGJAH3NFZLJMWQ:DO1PzrgSpA5aK+5ogXIfocl+qkY= 2017-03-14@07:41:54.474 D [validate-34 ] headers:logDebug - >> Content-Type: application/unknown 2017-03-14@07:41:54.475 D [validate-34 ] headers:logDebug - >> Content-Length: 1024 2017-03-14@07:41:54.710 D [validate-34 ] pCommandExecutorService:logDebug - Receiving response 1571963586: HTTP/1.1 200 OK *2017-03-14@07:41:54.710 D [validate-34 ] headers:logDebug - << HTTP/1.1 200 OK* 2017-03-14@07:41:54.711 D [validate-34 ] headers:logDebug - << Server: AmazonS3 2*017-03-14@07:41:54.711 D [validate-34 ] headers:logDebug - << ETag: "c9a34cfc85d982698c6ac89f76071abd"* 2017-03-14@07:41:54.711 D [validate-34 ] headers:logDebug - << x-amz-request-id: FEAF745B7A6ADCCA 2017-03-14@07:41:54.711 D [validate-34 ] headers:logDebug - << x-amz-id-2: PoXZwaaoyJdgN4IcWBPFindUVfFBUl51zbQVQ2iL8YHDosyVJksqE3JotpiwYtMmUQDxjADzBVA= 2017-03-14@07:41:54.711 D [validate-34 ] headers:logDebug - << Date: Tue, 14 Mar 2017 11:41:54 GMT 2017-03-14@07:41:54.712 D [validate-34 ] headers:logDebug - << Content-Type: application/unknown 2017-03-14@07:41:54.712 D [validate-34 ] headers:logDebug - << Content-Length: 0 2017-03-14@07:47:44.733 D [validate-41 ] pCommandExecutorService:logDebug - Sending request -225160499: GET https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com/IOPS-4 HTTP/1.1 2017-03-14@07:47:44.733 D [validate-41 ] headers:logDebug - >> GET https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com/IOPS-4 HTTP/1.1 2017-03-14@07:47:44.733 D [validate-41 ] headers:logDebug - >> Host: mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3.amazonaws.com 2017-03-14@07:47:44.733 D [validate-41 ] headers:logDebug - >> Date: Tue, 14 Mar 2017 11:47:42 GMT 2017-03-14@07:47:44.733 D [validate-41 ] headers:logDebug - >> Authorization: AWS AKIAJFXGJAH3NFZLJMWQ:iAI8UP3HnovXs3/4F6vqJWn/DQw= 2017-03-14@07:47:44.785 D [validate-41 ] pCommandExecutorService:logDebug - Receiving response -225160499: HTTP/1.1 307 Temporary Redirect 2017-03-14@07:47:44.785 D [validate-41 ] headers:logDebug - << HTTP/1.1 307 Temporary Redirect 2017-03-14@07:47:44.785 D [validate-41 ] headers:logDebug - << Transfer-Encoding: chunked 2017-03-14@07:47:44.785 D [validate-41 ] headers:logDebug - << Server: AmazonS3 2017-03-14@07:47:44.785 D [validate-41 ] headers:logDebug - << x-amz-request-id: F76A3D3964A41BA2 2017-03-14@07:47:44.786 D [validate-41 ] headers:logDebug - << x-amz-id-2: 7FrHX+IorCrvzJ4JKNi7DzT6JorA/mYcDnxmgJKLAtoDtwxB888AiYyGFE2Y6/PjOameoA38YvI= 2017-03-14@07:47:44.786 D [validate-41 ] headers:logDebug - << Date: Tue, 14 Mar 2017 11:47:42 GMT 2017-03-14@07:47:44.786 D [validate-41 ] headers:logDebug - << x-amz-bucket-region: ap-northeast-1 2017-03-14@07:47:44.786 D [validate-41 ] headers:logDebug - << Location: https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4 2017-03-14@07:47:44.786 D [validate-41 ] headers:logDebug - << Content-Type: application/xml 2017-03-14@07:47:44.786 D [validate-41 ] wire:logDebug - << "<?xml version="1.0" encoding="UTF-8"?>[\n]" 2017-03-14@07:47:44.786 D [validate-41 ] wire:logDebug - << "<Error><Code>TemporaryRedirect</Code><Message>Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.</Message><Bucket>mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918</Bucket><Endpoint> mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com </Endpoint><RequestId>F76A3D3964A41BA2</RequestId><HostId>7FrHX+IorCrvzJ4JKNi7DzT6JorA/mYcDnxmgJKLAtoDtwxB888AiYyGFE2Y6/PjOameoA38YvI=</HostId></Error>" *2017-03-14@07:47:44.787 D [validate-41 ] pCommandExecutorService:logDebug - Sending request -476513873: GET https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4 <https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4> HTTP/1.1* *2017-03-14@07:47:44.787 D [validate-41 ] headers:logDebug - >> GET https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4 <https://mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com/IOPS-4> HTTP/1.1* 2017-03-14@07:47:44.787 D [validate-41 ] headers:logDebug - >> Host: mcstore-ttv-validator-4ca2d424-1c7e-44e7-b83f-7d0e45706918.s3-ap-northeast-1.amazonaws.com 2017-03-14@07:47:44.787 D [validate-41 ] headers:logDebug - >> Date: Tue, 14 Mar 2017 11:47:42 GMT 2017-03-14@07:47:44.787 D [validate-41 ] headers:logDebug - >> Authorization: AWS AKIAJFXGJAH3NFZLJMWQ:iAI8UP3HnovXs3/4F6vqJWn/DQw= 2017-03-14@07:47:45.260 D [validate-41 ] pCommandExecutorService:logDebug - Receiving response -476513873: HTTP/1.1 404 Not Found 2017-03-14@07:47:45.260 D [validate-41 ] headers:logDebug - << HTTP/1.1 404 Not Found 2017-03-14@07:47:45.260 D [validate-41 ] headers:logDebug - << Transfer-Encoding: chunked 2017-03-14@07:47:45.260 D [validate-41 ] headers:logDebug - << Server: AmazonS3 2017-03-14@07:47:45.261 D [validate-41 ] headers:logDebug - << x-amz-request-id: 5A08FE7C9A9FDADA 2017-03-14@07:47:45.261 D [validate-41 ] headers:logDebug - << x-amz-id-2: gTbzpFjMIhD/P0QO6AcEAiqKo8WKHAs5olmHz5fegMCbfMnSCbGvnuCcmHmx8qrFF6fkdKRI38o= 2017-03-14@07:47:45.261 D [validate-41 ] headers:logDebug - << Date: Tue, 14 Mar 2017 11:47:42 GMT 2017-03-14@07:47:45.261 D [validate-41 ] headers:logDebug - << Content-Type: application/xml 2017-03-14@07:47:45.261 D [validate-41 ] wire:logDebug - << "<?xml version="1.0" encoding="UTF-8"?>[\n]" *2017-03-14@07:47:45.261 D [validate-41 ] wire:logDebug - << "<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>IOPS-4</Key><RequestId>5A08FE7C9A9FDADA</RequestId><HostId>gTbzpFjMIhD/P0QO6AcEAiqKo8WKHAs5olmHz5fegMCbfMnSCbGvnuCcmHmx8qrFF6fkdKRI38o=</HostId></Error>"* Thanks, Arunagiri
