[
https://issues.apache.org/jira/browse/HADOOP-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157950#comment-16157950
]
Aaron Fabbri commented on HADOOP-14738:
---------------------------------------
First off, removing a bunch of older code brings me joy. Thanks for that.
This looks good overall.
The stub s3n FS is a nice thing to do for users. I tried it out but just got
this:
{noformat}
$ hadoop fs -ls s3n://fabbri-dev/
2017-09-07 17:34:32,867 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
ls: No FileSystem for scheme "s3n"
{noformat}
It does work if I use {{s3a://}} though.
{noformat}
--- a/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties\
...
-log4j.logger.com.amazonaws=ERROR
-log4j.logger.com.amazonaws.http.AmazonHttpClient=ERROR
-log4j.logger.org.apache.hadoop.fs.s3a.S3AFileSystem=WARN
+#log4j.logger.com.amazonaws=ERROR
+#log4j.logger.com.amazonaws.http.AmazonHttpClient=ERROR
+#log4j.logger.org.apache.hadoop.fs.s3a.S3AFileSystem=WARN
{noformat}
Does this change defaults to INFO?
{noformat}
+S3N wasa Hadoop filesystem client which can read or write data stored
+in Amazon S3. It uses URLs with the schema `s3n://`.
{noformat}
I like the migration document. Small typo here "wasa".
{noformat}
+ hdfs fs -ls s3a://my-bucket/
...
+means of verifying that the key and secret can access a bucket;
+
+ hdfs fs -ls s3a://key:secret@my-bucket/
{noformat}
Probably existing stuff, but should this be `hadoop fs` or `hdfs dfs`?
I applied the patch and will go ahead and run all tests in hadoop-aws.
> Remove S3N and obsolete bits of S3A; rework docs
> ------------------------------------------------
>
> Key: HADOOP-14738
> URL: https://issues.apache.org/jira/browse/HADOOP-14738
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0, 3.0.0-beta1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Blocker
> Attachments: HADOOP-14738-002.patch, HADOOP-14738-003.patch,
> HADOOP-14738-004.patch, HADOOP-14739-001.patch
>
>
> We are all happy with S3A; it's been stable since Hadoop 2.7 and high-perf
> since Hadoop 2.8
> It's now time to kill S3N off, remove the source, the tests, the transitive
> dependencies. This patch does that.
> It also removes the obsolete, original s3a output stream; the fast/block
> upload stream has been stable and is much more manageable and maintained (put
> differently: we don't ever look at the original S3A output stream, and tell
> people not to use it for performance reasons).
> As well as cutting the features, this patch updates the aws docs with
> * split out s3n migration page
> * split out troubleshooting page
> * rework of the "uploading data with s3a" section of index.md, as there's no
> need to discuss the slow upload except in the past tense...all that is needed
> is to list the buffering and thread tuning options of the block uploader.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
