[
https://issues.apache.org/jira/browse/HADOOP-14140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14140:
------------------------------------
Description:
Hi:
UPDATE: This stack trace is being caused because there are 0 objects in the
target directory. S3A is unable to handle 0 objects
---
Connecting S3A to a 3rd party object store does not work. This is a publicly
hosted grid and i can provide credentials if required. Please see the debug log
below
There are two problems -
1. Path Style setting is ignored, and S3A always uses host style addressing
2. Even when host style is specified, it is unable to proceed, see debug log
(in comment below)
was:
Hi:
UPDATE: This stack trace is being caused because there are 0 objects in the
target directory. S3A is unable to handle 0 objects
---
Connecting S3A to a 3rd party object store does not work. This is a publicly
hosted grid and i can provide credentials if required. Please see the debug log
below
There are two problems -
1. Path Style setting is ignored, and S3A always uses host style addressing
2. Even when host style is specified, it is unable to proceed, see debug log
17/03/02 13:35:03 DEBUG HadoopRDD: Creating new JobConf and caching it for
later re-use
17/03/02 13:35:03 DEBUG InternalConfig: Configuration override
awssdk_config_override.json not found.
17/03/02 13:35:03 DEBUG AWSCredentialsProviderChain: Loading credentials from
BasicAWSCredentialsProvider
17/03/02 13:35:03 DEBUG S3Signer: Calculated string to sign:
"HEAD
application/x-www-form-urlencoded; charset=utf-8
Thu, 02 Mar 2017 21:35:03 GMT
/solidfire/"
17/03/02 13:35:03 DEBUG request: Sending Request: HEAD
https://solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082 / Headers:
(Authorization: AWS 2SNAJYEMQU45YPVYC89D:WO0R+mPeYoQ2V29L4dMUJSSSVsQ=,
User-Agent: aws-sdk-java/1.7.4 Mac_OS_X/10.12.3
Java_HotSpot(TM)_64-Bit_Server_VM/25.60-b23/1.8.0_60, Date: Thu, 02 Mar 2017
21:35:03 GMT, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
17/03/02 13:35:03 DEBUG PoolingClientConnectionManager: Connection request:
[route:
{s}->https://solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082][total
kept alive: 0; route allocated: 0 of 15; total allocated: 0 of 15]
17/03/02 13:35:03 DEBUG PoolingClientConnectionManager: Connection leased: [id:
0][route:
{s}->https://solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082][total
kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15]
17/03/02 13:35:03 DEBUG DefaultClientConnectionOperator: Connecting to
solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082
17/03/02 13:35:03 DEBUG RequestAddCookies: CookieSpec selected: default
17/03/02 13:35:03 DEBUG RequestAuthCache: Auth cache not set in the context
17/03/02 13:35:03 DEBUG RequestProxyAuthentication: Proxy auth state:
UNCHALLENGED
17/03/02 13:35:03 DEBUG SdkHttpClient: Attempt 1 to execute request
17/03/02 13:35:03 DEBUG DefaultClientConnection: Sending request: HEAD /
HTTP/1.1
17/03/02 13:35:03 DEBUG wire: >> "HEAD / HTTP/1.1[\r][\n]"
17/03/02 13:35:03 DEBUG wire: >> "Host:
solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082[\r][\n]"
17/03/02 13:35:03 DEBUG wire: >> "Authorization: AWS
2SNAJYEMQU45YPVYC89D:WO0R+mPeYoQ2V29L4dMUJSSSVsQ=[\r][\n]"
17/03/02 13:35:03 DEBUG wire: >> "User-Agent: aws-sdk-java/1.7.4
Mac_OS_X/10.12.3 Java_HotSpot(TM)_64-Bit_Server_VM/25.60-b23/1.8.0_60[\r][\n]"
17/03/02 13:35:03 DEBUG wire: >> "Date: Thu, 02 Mar 2017 21:35:03 GMT[\r][\n]"
17/03/02 13:35:03 DEBUG wire: >> "Content-Type:
application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
17/03/02 13:35:03 DEBUG wire: >> "Connection: Keep-Alive[\r][\n]"
17/03/02 13:35:03 DEBUG wire: >> "[\r][\n]"
17/03/02 13:35:03 DEBUG headers: >> HEAD / HTTP/1.1
17/03/02 13:35:03 DEBUG headers: >> Host:
solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082
17/03/02 13:35:03 DEBUG headers: >> Authorization: AWS
2SNAJYEMQU45YPVYC89D:WO0R+mPeYoQ2V29L4dMUJSSSVsQ=
17/03/02 13:35:03 DEBUG headers: >> User-Agent: aws-sdk-java/1.7.4
Mac_OS_X/10.12.3 Java_HotSpot(TM)_64-Bit_Server_VM/25.60-b23/1.8.0_60
17/03/02 13:35:03 DEBUG headers: >> Date: Thu, 02 Mar 2017 21:35:03 GMT
17/03/02 13:35:03 DEBUG headers: >> Content-Type:
application/x-www-form-urlencoded; charset=utf-8
17/03/02 13:35:03 DEBUG headers: >> Connection: Keep-Alive
17/03/02 13:35:03 DEBUG wire: << "HTTP/1.1 200 OK[\r][\n]"
17/03/02 13:35:03 DEBUG wire: << "Date: Thu, 02 Mar 2017 21:35:03 GMT[\r][\n]"
17/03/02 13:35:03 DEBUG wire: << "Connection: KEEP-ALIVE[\r][\n]"
17/03/02 13:35:03 DEBUG wire: << "Server: StorageGRID/10.3.0.1[\r][\n]"
17/03/02 13:35:03 DEBUG wire: << "x-amz-request-id: 640939184[\r][\n]"
17/03/02 13:35:03 DEBUG wire: << "Content-Length: 0[\r][\n]"
17/03/02 13:35:03 DEBUG wire: << "[\r][\n]"
17/03/02 13:35:03 DEBUG DefaultClientConnection: Receiving response: HTTP/1.1
200 OK
17/03/02 13:35:03 DEBUG headers: << HTTP/1.1 200 OK
17/03/02 13:35:03 DEBUG headers: << Date: Thu, 02 Mar 2017 21:35:03 GMT
17/03/02 13:35:03 DEBUG headers: << Connection: KEEP-ALIVE
17/03/02 13:35:03 DEBUG headers: << Server: StorageGRID/10.3.0.1
17/03/02 13:35:03 DEBUG headers: << x-amz-request-id: 640939184
17/03/02 13:35:03 DEBUG headers: << Content-Length: 0
17/03/02 13:35:03 DEBUG SdkHttpClient: Connection can be kept alive indefinitely
17/03/02 13:35:04 DEBUG PoolingClientConnectionManager: Connection [id:
0][route: {s}->https://solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082]
can be kept alive indefinitely
17/03/02 13:35:04 DEBUG PoolingClientConnectionManager: Connection released:
[id: 0][route:
{s}->https://solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082][total
kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
17/03/02 13:35:04 DEBUG request: Received successful response: 200, AWS Request
ID: 640939184
17/03/02 13:35:04 DEBUG S3AFileSystem: Getting path status for s3a://solidfire/
()
17/03/02 13:35:04 DEBUG S3Signer: Calculated string to sign:
"GET
application/x-www-form-urlencoded; charset=utf-8
Thu, 02 Mar 2017 21:35:04 GMT
/solidfire/"
17/03/02 13:35:04 DEBUG request: Sending Request: GET
https://solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082 / Parameters:
(max-keys: 1, prefix: , delimiter: /, ) Headers: (Authorization: AWS
2SNAJYEMQU45YPVYC89D:lfPlFQy/vzQuLHXisTmL1Bb5D/k=, User-Agent:
aws-sdk-java/1.7.4 Mac_OS_X/10.12.3
Java_HotSpot(TM)_64-Bit_Server_VM/25.60-b23/1.8.0_60, Date: Thu, 02 Mar 2017
21:35:04 GMT, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
17/03/02 13:35:04 DEBUG PoolingClientConnectionManager: Connection request:
[route:
{s}->https://solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082][total
kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
17/03/02 13:35:04 DEBUG PoolingClientConnectionManager: Connection leased: [id:
0][route:
{s}->https://solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082][total
kept alive: 0; route allocated: 1 of 15; total allocated: 1 of 15]
17/03/02 13:35:04 DEBUG SdkHttpClient: Stale connection check
17/03/02 13:35:04 DEBUG RequestAddCookies: CookieSpec selected: default
17/03/02 13:35:04 DEBUG RequestAuthCache: Auth cache not set in the context
17/03/02 13:35:04 DEBUG RequestProxyAuthentication: Proxy auth state:
UNCHALLENGED
17/03/02 13:35:04 DEBUG SdkHttpClient: Attempt 1 to execute request
17/03/02 13:35:04 DEBUG DefaultClientConnection: Sending request: GET
/?max-keys=1&prefix=&delimiter=%2F HTTP/1.1
17/03/02 13:35:04 DEBUG wire: >> "GET /?max-keys=1&prefix=&delimiter=%2F
HTTP/1.1[\r][\n]"
17/03/02 13:35:04 DEBUG wire: >> "Host:
solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082[\r][\n]"
17/03/02 13:35:04 DEBUG wire: >> "Authorization: AWS
2SNAJYEMQU45YPVYC89D:lfPlFQy/vzQuLHXisTmL1Bb5D/k=[\r][\n]"
17/03/02 13:35:04 DEBUG wire: >> "User-Agent: aws-sdk-java/1.7.4
Mac_OS_X/10.12.3 Java_HotSpot(TM)_64-Bit_Server_VM/25.60-b23/1.8.0_60[\r][\n]"
17/03/02 13:35:04 DEBUG wire: >> "Date: Thu, 02 Mar 2017 21:35:04 GMT[\r][\n]"
17/03/02 13:35:04 DEBUG wire: >> "Content-Type:
application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
17/03/02 13:35:04 DEBUG wire: >> "Connection: Keep-Alive[\r][\n]"
17/03/02 13:35:04 DEBUG wire: >> "[\r][\n]"
17/03/02 13:35:04 DEBUG headers: >> GET /?max-keys=1&prefix=&delimiter=%2F
HTTP/1.1
17/03/02 13:35:04 DEBUG headers: >> Host:
solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082
17/03/02 13:35:04 DEBUG headers: >> Authorization: AWS
2SNAJYEMQU45YPVYC89D:lfPlFQy/vzQuLHXisTmL1Bb5D/k=
17/03/02 13:35:04 DEBUG headers: >> User-Agent: aws-sdk-java/1.7.4
Mac_OS_X/10.12.3 Java_HotSpot(TM)_64-Bit_Server_VM/25.60-b23/1.8.0_60
17/03/02 13:35:04 DEBUG headers: >> Date: Thu, 02 Mar 2017 21:35:04 GMT
17/03/02 13:35:04 DEBUG headers: >> Content-Type:
application/x-www-form-urlencoded; charset=utf-8
17/03/02 13:35:04 DEBUG headers: >> Connection: Keep-Alive
17/03/02 13:35:04 DEBUG wire: << "HTTP/1.1 200 OK[\r][\n]"
17/03/02 13:35:04 DEBUG wire: << "Date: Thu, 02 Mar 2017 21:35:04 GMT[\r][\n]"
17/03/02 13:35:04 DEBUG wire: << "Connection: KEEP-ALIVE[\r][\n]"
17/03/02 13:35:04 DEBUG wire: << "Server: StorageGRID/10.3.0.1[\r][\n]"
17/03/02 13:35:04 DEBUG wire: << "x-amz-request-id: 640939184[\r][\n]"
17/03/02 13:35:04 DEBUG wire: << "Content-Length: 256[\r][\n]"
17/03/02 13:35:04 DEBUG wire: << "Content-Type: application/xml[\r][\n]"
17/03/02 13:35:04 DEBUG wire: << "[\r][\n]"
17/03/02 13:35:04 DEBUG DefaultClientConnection: Receiving response: HTTP/1.1
200 OK
17/03/02 13:35:04 DEBUG headers: << HTTP/1.1 200 OK
17/03/02 13:35:04 DEBUG headers: << Date: Thu, 02 Mar 2017 21:35:04 GMT
17/03/02 13:35:04 DEBUG headers: << Connection: KEEP-ALIVE
17/03/02 13:35:04 DEBUG headers: << Server: StorageGRID/10.3.0.1
17/03/02 13:35:04 DEBUG headers: << x-amz-request-id: 640939184
17/03/02 13:35:04 DEBUG headers: << Content-Length: 256
17/03/02 13:35:04 DEBUG headers: << Content-Type: application/xml
17/03/02 13:35:04 DEBUG SdkHttpClient: Connection can be kept alive indefinitely
17/03/02 13:35:04 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined
for handler class
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
17/03/02 13:35:04 DEBUG wire: << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
17/03/02 13:35:04 DEBUG wire: << "<ListBucketResult
xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Name>solidfire</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated></ListBucketResult>"
17/03/02 13:35:04 DEBUG PoolingClientConnectionManager: Connection [id:
0][route: {s}->https://solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082]
can be kept alive indefinitely
17/03/02 13:35:04 DEBUG PoolingClientConnectionManager: Connection released:
[id: 0][route:
{s}->https://solidfire.vmasgwwebg01-tst.webscaledemo.netapp.com:8082][total
kept alive: 1; route allocated: 1 of 15; total allocated: 1 of 15]
17/03/02 13:35:04 DEBUG XmlResponsesSaxParser: Parsing XML response document
with handler: class
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
17/03/02 13:35:04 DEBUG XmlResponsesSaxParser: Examining listing for bucket:
solidfire
17/03/02 13:35:04 DEBUG request: Received successful response: 200, AWS Request
ID: 640939184
17/03/02 13:35:04 DEBUG S3AFileSystem: Not Found: s3a://solidfire/
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
s3a://solidfire/
at
org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1958)
at org.apache.spark.rdd.RDD.count(RDD.scala:1157)
... 53 elided
> S3A reporting "Not found on empty bucket
> ----------------------------------------
>
> Key: HADOOP-14140
> URL: https://issues.apache.org/jira/browse/HADOOP-14140
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.7.3
> Environment: Non-AWS s3 implementation
> Reporter: Vishnu Vardhan
> Priority: Minor
>
> Hi:
> UPDATE: This stack trace is being caused because there are 0 objects in the
> target directory. S3A is unable to handle 0 objects
> ---
> Connecting S3A to a 3rd party object store does not work. This is a publicly
> hosted grid and i can provide credentials if required. Please see the debug
> log below
> There are two problems -
> 1. Path Style setting is ignored, and S3A always uses host style addressing
> 2. Even when host style is specified, it is unable to proceed, see debug log
> (in comment below)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
