ConfX created HADOOP-18804:
------------------------------
Summary: Out of Memory when mistakenly set io.file.buffer.size to
a large number
Key: HADOOP-18804
URL: https://issues.apache.org/jira/browse/HADOOP-18804
Project: Hadoop Common
Issue Type: Bug
Components: common, fs
Reporter: ConfX
h2. What happened:
When setting io.file.buffer.size to a large number,
BufferedIOStatisticsOutputStream in Hcommon throws an out-of-memory exception
due to inappropriate checking and handling.
The config is used to initialize a file system by passing it as one of the
parameters bufferSize.
h2. Buggy code:
In RawLocalFileSystem.java
{code:java}
private FSDataOutputStream create(Path f, boolean overwrite,
boolean createParent, int bufferSize, short replication, long blockSize,
Progressable progress, FsPermission permission) throws IOException {
...
return new FSDataOutputStream(new BufferedIOStatisticsOutputStream(
createOutputStreamWithMode(f, false, permission), bufferSize, true), <<---
creates a BufferedIOStatisticsOutputStream with bufferSize, often set to config
io.file.buffer.size
statistics);
} {code}
In BufferedIOStatisticsOutputStream.java:
{code:java}
public class BufferedIOStatisticsOutputStream extends BufferedOutputStream
implements IOStatisticsSource, Syncable, StreamCapabilities {
...
public BufferedIOStatisticsOutputStream(
final OutputStream out,
final int size,
final boolean downgradeSyncable) {
super(out, size); <<--- init the BufferedOutputStream with a huge buffer
size
...
}{code}
h2. StackTrace:
{code:java}
java.lang.OutOfMemoryError: Java heap space
at
java.base/java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:75)
at
org.apache.hadoop.fs.statistics.BufferedIOStatisticsOutputStream.<init>(BufferedIOStatisticsOutputSt
ream.java:78)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:428)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:413)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)
at
org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset(ContractTestUtils.java:183)
at
org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset(ContractTestUtils.java:152)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.expectRenameUnderFileFails(AbstractContract
RenameTest.java:335)
...{code}
h2. Reproduce:
(1) Set io.file.buffer.size to a large value, e.g., 2112001717
(2) Run a simple test that exercises this parameter, e.g.
org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractRename#testRenameFileUnderFile
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
