[
https://issues.apache.org/jira/browse/HADOOP-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15892196#comment-15892196
]
Steve Loughran commented on HADOOP-14138:
-----------------------------------------
Did run one test with Filesystem log to DEBUG, so have it print what goes on
with discovery.
s3 and s3n are coming in from service loader, s3a is coming in from the
core-default file. It probably always did, given that entry existed ... that
service load has always been unneeded
{code}
2017-03-02 12:54:07,050 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:loadFileSystems(3147)) - Loading filesystems
2017-03-02 12:54:07,057 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:loadFileSystems(3159)) - s3:// = class
org.apache.hadoop.fs.s3.S3FileSystem from null
2017-03-02 12:54:07,062 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:loadFileSystems(3159)) - s3n:// = class
org.apache.hadoop.fs.s3native.NativeS3FileSystem from null
2017-03-02 12:54:07,070 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:loadFileSystems(3159)) - file:// = class
org.apache.hadoop.fs.LocalFileSystem from
/Users/stevel/.m2/repository/org/apache/hadoop/hadoop-common/2.9.0-SNAPSHOT/hadoop-common-2.9.0-SNAPSHOT.jar
2017-03-02 12:54:07,075 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:loadFileSystems(3159)) - viewfs:// = class
org.apache.hadoop.fs.viewfs.ViewFileSystem from
/Users/stevel/.m2/repository/org/apache/hadoop/hadoop-common/2.9.0-SNAPSHOT/hadoop-common-2.9.0-SNAPSHOT.jar
2017-03-02 12:54:07,077 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:loadFileSystems(3159)) - ftp:// = class
org.apache.hadoop.fs.ftp.FTPFileSystem from
/Users/stevel/.m2/repository/org/apache/hadoop/hadoop-common/2.9.0-SNAPSHOT/hadoop-common-2.9.0-SNAPSHOT.jar
2017-03-02 12:54:07,079 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:loadFileSystems(3159)) - har:// = class
org.apache.hadoop.fs.HarFileSystem from
/Users/stevel/.m2/repository/org/apache/hadoop/hadoop-common/2.9.0-SNAPSHOT/hadoop-common-2.9.0-SNAPSHOT.jar
2017-03-02 12:54:07,085 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:loadFileSystems(3159)) - hdfs:// = class
org.apache.hadoop.hdfs.DistributedFileSystem from
/Users/stevel/.m2/repository/org/apache/hadoop/hadoop-hdfs-client/2.9.0-SNAPSHOT/hadoop-hdfs-client-2.9.0-SNAPSHOT.jar
2017-03-02 12:54:07,201 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:loadFileSystems(3159)) - webhdfs:// = class
org.apache.hadoop.hdfs.web.WebHdfsFileSystem from
/Users/stevel/.m2/repository/org/apache/hadoop/hadoop-hdfs-client/2.9.0-SNAPSHOT/hadoop-hdfs-client-2.9.0-SNAPSHOT.jar
2017-03-02 12:54:07,202 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:loadFileSystems(3159)) - swebhdfs:// = class
org.apache.hadoop.hdfs.web.SWebHdfsFileSystem from
/Users/stevel/.m2/repository/org/apache/hadoop/hadoop-hdfs-client/2.9.0-SNAPSHOT/hadoop-hdfs-client-2.9.0-SNAPSHOT.jar
2017-03-02 12:54:07,206 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:loadFileSystems(3159)) - hftp:// = class
org.apache.hadoop.hdfs.web.HftpFileSystem from
/Users/stevel/.m2/repository/org/apache/hadoop/hadoop-hdfs-client/2.9.0-SNAPSHOT/hadoop-hdfs-client-2.9.0-SNAPSHOT.jar
2017-03-02 12:54:07,206 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:loadFileSystems(3159)) - hsftp:// = class
org.apache.hadoop.hdfs.web.HsftpFileSystem from
/Users/stevel/.m2/repository/org/apache/hadoop/hadoop-hdfs-client/2.9.0-SNAPSHOT/hadoop-hdfs-client-2.9.0-SNAPSHOT.jar
2017-03-02 12:54:07,206 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:getFileSystemClass(3202)) - Looking for FS supporting s3a
2017-03-02 12:54:07,206 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:getFileSystemClass(3206)) - looking for configuration option
fs.s3a.impl
2017-03-02 12:54:07,250 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:getFileSystemClass(3216)) - Filesystem s3a defined in
configuration option
2017-03-02 12:54:07,251 [Thread-0] DEBUG fs.FileSystem
(FileSystem.java:getFileSystemClass(3222)) - FS for s3a is class
org.apache.hadoop.fs.s3a.S3AFileSystem
// and the actual test itself
2017-03-02 12:54:08,231 [Thread-0] INFO contract.AbstractFSContractTestBase
(AbstractFSContractTestBase.java:setup(184)) - Test filesystem =
s3a://hwdev-steve-ireland-new implemented by
S3AFileSystem{uri=s3a://hwdev-steve-ireland-new,
workingDir=s3a://hwdev-steve-ireland-new/user/stevel, inputPolicy=normal,
partSize=8000000, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536,
blockSize=33554432, multiPartThreshold=2147483647,
serverSideEncryptionAlgorithm='SSE_S3',
blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@be1e9c6,
boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=30,
available=30, waiting=0}, activeCount=0},
unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@503f1b5a[Running,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0],
statistics {0 bytes read, 0 bytes written, 0 read ops, 0 large read ops, 0
write ops}, metrics {{Context=S3AFileSystem}
{FileSystemId=1408be8b-d462-4388-91fa-90b4022356fc-hwdev-steve-ireland-new}
{fsURI=s3a://hwdev-steve-ireland-new/} {files_created=0} {files_copied=0}
{files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=0}
{directories_created=0} {directories_deleted=0} {ignored_errors=0}
{op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=0}
{op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0}
{op_list_located_status=0} {op_list_status=0} {op_mkdirs=0} {op_rename=0}
{object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0}
{object_continue_list_requests=0} {object_metadata_requests=0}
{object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0}
{object_put_requests_completed=0} {stream_write_failures=0}
{stream_write_block_uploads=0} {stream_write_block_uploads_committed=0}
{stream_write_block_uploads_aborted=0} {stream_write_total_time=0}
{stream_write_total_data=0} {object_put_requests_active=0}
{object_put_bytes_pending=0} {stream_write_block_uploads_active=0}
{stream_write_block_uploads_pending=0}
{stream_write_block_uploads_data_pending=0} {stream_read_fully_operations=0}
{stream_opened=0} {stream_bytes_skipped_on_seek=0} {stream_closed=0}
{stream_bytes_backwards_on_seek=0} {stream_bytes_read=0}
{stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0}
{stream_close_operations=0} {stream_read_operations=0} {stream_aborted=0}
{stream_forward_seek_operations=0} {stream_backward_seek_operations=0}
{stream_seek_operations=0} {stream_bytes_read_in_close=0}
{stream_read_exceptions=0} }}
2017-03-02 12:54:08,628 [JUnit-testCreateNonRecursiveSuccess] DEBUG
fs.FileSystem (FileSystem.java:getFileSystemClass(3202)) - Looking for FS
supporting file
2017-03-02 12:54:08,628 [JUnit-testCreateNonRecursiveSuccess] DEBUG
fs.FileSystem (FileSystem.java:getFileSystemClass(3206)) - looking for
configuration option fs.file.impl
2017-03-02 12:54:08,628 [JUnit-testCreateNonRecursiveSuccess] DEBUG
fs.FileSystem (FileSystem.java:getFileSystemClass(3213)) - Looking in service
filesystems for implementation class
2017-03-02 12:54:08,628 [JUnit-testCreateNonRecursiveSuccess] DEBUG
fs.FileSystem (FileSystem.java:getFileSystemClass(3222)) - FS for file is class
org.apache.hadoop.fs.LocalFileSystem
2017-03-02 12:54:09,392 [JUnit-testCreateNonRecursiveSuccess] INFO
contract.AbstractFSContractTestBase
(AbstractFSContractTestBase.java:describe(255)) - closing file system
{code}
> Remove S3A ref from META-INF service discovery, rely on existing core-default
> entry
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-14138
> URL: https://issues.apache.org/jira/browse/HADOOP-14138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Critical
> Attachments: HADOOP-14138-branch-2-001.patch
>
>
> As discussed in HADOOP-14132, the shaded AWS library is killing performance
> starting all hadoop operations, due to classloading on FS service discovery.
> This is despite the fact that there is an entry for fs.s3a.impl in
> core-default.xml, *we don't need service discovery here*
> Proposed:
> # cut the entry from
> {/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem}}
> # when HADOOP-14132 is in, move to that, including declaring an XML file
> exclusively for s3a entries
> I want this one in first as its a major performance regression, and one we
> coula actually backport to 2.7.x, just to improve load time slightly there too
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
