[
https://issues.apache.org/jira/browse/HADOOP-18804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ConfX updated HADOOP-18804:
---------------------------
Attachment: reproduce.sh
> Out of Memory when mistakenly set io.file.buffer.size to a large number
> -----------------------------------------------------------------------
>
> Key: HADOOP-18804
> URL: https://issues.apache.org/jira/browse/HADOOP-18804
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs
> Reporter: ConfX
> Priority: Critical
> Attachments: reproduce.sh
>
>
> h2. What happened:
> When setting io.file.buffer.size to a large number,
> BufferedIOStatisticsOutputStream in Hcommon throws an out-of-memory exception
> due to inappropriate checking and handling.
> The config is used to initialize a file system by passing it as one of the
> parameters bufferSize.
> h2. Buggy code:
> In RawLocalFileSystem.java
> {code:java}
> private FSDataOutputStream create(Path f, boolean overwrite,
> boolean createParent, int bufferSize, short replication, long blockSize,
> Progressable progress, FsPermission permission) throws IOException {
> ...
> return new FSDataOutputStream(new BufferedIOStatisticsOutputStream(
> createOutputStreamWithMode(f, false, permission), bufferSize, true),
> <<--- creates a BufferedIOStatisticsOutputStream with bufferSize, often set
> to config io.file.buffer.size
> statistics);
> } {code}
> In BufferedIOStatisticsOutputStream.java:
> {code:java}
> public class BufferedIOStatisticsOutputStream extends BufferedOutputStream
> implements IOStatisticsSource, Syncable, StreamCapabilities {
> ...
> public BufferedIOStatisticsOutputStream(
> final OutputStream out,
> final int size,
> final boolean downgradeSyncable) {
> super(out, size); <<--- init the BufferedOutputStream with a huge buffer
> size
> ...
> }{code}
> h2. StackTrace:
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> at
> java.base/java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:75)
> at
> org.apache.hadoop.fs.statistics.BufferedIOStatisticsOutputStream.<init>(BufferedIOStatisticsOutputSt
> ream.java:78)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:428)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:413)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset(ContractTestUtils.java:183)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset(ContractTestUtils.java:152)
> at
> org.apache.hadoop.fs.contract.AbstractContractRenameTest.expectRenameUnderFileFails(AbstractContract
> RenameTest.java:335)
> ...{code}
> h2. Reproduce:
> (1) Set io.file.buffer.size to a large value, e.g., 2112001717
> (2) Run a simple test that exercises this parameter, e.g.
> org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractRename#testRenameFileUnderFile
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
