Ted Yu created HADOOP-11446:
-------------------------------
Summary: S3AOutputStream should use shared thread pool to avoid
OutOfMemoryError
Key: HADOOP-11446
URL: https://issues.apache.org/jira/browse/HADOOP-11446
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Here is part of the output including the OOME when hbase snapshot is exported
to s3a (nofile ulimit was increased to 102400):
{code}
2014-12-19 13:15:03,895 INFO [main] s3a.S3AFileSystem: OutputStream for key
'FastQueryPOC/2014-12-11/EVENT1-IDX-snapshot/.hbase-snapshot/.tmp/EVENT1_IDX_snapshot_2012_12_11/
650a5678810fbdaa91809668d11ccf09/.regioninfo' closed. Now beginning upload
2014-12-19 13:15:03,895 INFO [main] s3a.S3AFileSystem: Minimum upload part
size: 16777216 threshold2147483647
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new
native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:713)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
at
com.amazonaws.services.s3.transfer.internal.UploadMonitor.<init>(UploadMonitor.java:129)
at
com.amazonaws.services.s3.transfer.TransferManager.upload(TransferManager.java:449)
at
com.amazonaws.services.s3.transfer.TransferManager.upload(TransferManager.java:382)
at
org.apache.hadoop.fs.s3a.S3AOutputStream.close(S3AOutputStream.java:127)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:54)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:356)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:356)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:791)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:882)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:886)
{code}
In S3AOutputStream#close():
{code}
TransferManager transfers = new TransferManager(client);
{code}
This results in each TransferManager creating its own thread pool, leading to
the OOME.
One solution is to pass shared thread pool to TransferManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
