Hello, I'd like to use swift storage provided by a ceph cluster (not under Openstack and no keystone installed) with hadoop (Ambari).
Version : HDP 2.4.0.0-169 HDFS 2.7.1.2. YARN 2.7.1.2.4 Ceph 10.2.2 (jewel) I'm able to connect to my swift bucket with rclone but I can't connect with hadoop, Id' like to know where I am wrong in my configuration. The command I run : > $ hadoop distcp \ > -D fs.swift.service.ceph.username=ymoulin:swift \ > -D fs.swift.service.ceph.password=<snip> \ > -D fs.swift.service.ceph.auth.url=http://node018.ceph.local:8080/auth/v1.0/ \ > -D fs.swift.service.ceph.http.port=80 \ > dcu.txt swift://mybigdatacontainer.ceph/dcu.txt > 16/06/29 09:58:35 ERROR tools.DistCp: Invalid arguments: > Invalid Response: Method POST on http://node018.ceph.local:8080/auth/v1.0/ > failed, status code: 405, status line: HTTP/1.1 405 Method Not Allowed POST > http://node018.ceph.local:8080/auth/v1.0/ => 405 : > {"Code":"MethodNotAllowed","RequestId":"tx0000000000000000e4505-0057737faa-19702-default","HostId":"19702-default-default"} > at > org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1501) > at > org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1402) > at > org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1079) > at > org.apache.hadoop.fs.swift.http.SwiftRestClient.authIfNeeded(SwiftRestClient.java:1298) > at > org.apache.hadoop.fs.swift.http.SwiftRestClient.preRemoteCommand(SwiftRestClient.java:1314) > at > org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1015) > at > org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.stat(SwiftNativeFileSystemStore.java:258) > at > org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:213) > at > org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:182) > at > org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) > at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:217) > at org.apache.hadoop.tools.DistCp.run(DistCp.java:116) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.hadoop.tools.DistCp.main(DistCp.java:430) > Invalid arguments: Method POST on http://node018.ceph.local:8080/auth/v1.0/ > failed, status code: 405, status line: HTTP/1.1 405 Method Not Allowed HTTP Headers from hadoop are : > POST /auth/v1.0/ HTTP/1.1 > User-Agent: Apache Hadoop Swift Client 2.7.1.2.4.0.0-169 from > 26104d8ac833884c8776473823007f176854f2eb by jenkins source checksum > cf48a4c63aaec76a714c1897e2ba8be6 > Host: node018.ceph.local:8080 > Content-Length: 133 > Content-Type: application/json; charset=UTF-8 > > {"auth":{"tenantName":null,"passwordCredentials":{"username":"ymoulin:swift","password":"<snip>"}}}HTTP/1.1 > 405 Method Not Allowed > x-amz-request-id: tx0000000000000000e4500-005772967d-19702-default > Content-Length: 123 > Accept-Ranges: bytes > Content-Type: application/json > Date: Tue, 28 Jun 2016 15:23:41 GMT > > {"Code":"MethodNotAllowed","RequestId":"tx0000000000000000e4500-005772967d-19702-default","HostId":"19702-default-default"} HTTP Headers from rclone are : > GET /auth/v1.0/ HTTP/1.1 > Host: node018.ceph.local:8080 > User-Agent: rclone/v1.29 > X-Auth-Key: <snip> > X-Auth-User: ymoulin:swift > Accept-Encoding: gzip > > HTTP/1.1 204 No Content > X-Storage-Url: http://node018.ceph.local:8080/swift/v1 > X-Storage-Token: > AUTH_rgwtk0d000000796d6f756c696e3a7377696674113cf055a759d9ec04e87357f1a0202e53c2b24a93c90be7eb58def247434d259c1c512b > X-Auth-Token: > AUTH_rgwtk0d000000796d6f756c696e3a7377696674113cf055a759d9ec04e87357f1a0202e53c2b24a93c90be7eb58def247434d259c1c512b > x-amz-request-id: tx0000000000000000e4501-0057729684-19702-default > Date: Tue, 28 Jun 2016 15:23:48 GMT Is there other parameter I should add to the hadoop command line ? Thanks for you help -- Yoann Moulin EPFL IC-IT --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
