[
https://issues.apache.org/jira/browse/HADOOP-18835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Smith Cruise updated HADOOP-18835:
----------------------------------
Environment:
I configure hedged read about below config:
```xml
<configuration>
<property>
<name>dfs.client.hedged.read.threadpool.size</name>
<value>128</value>
</property>
<property>
<name>dfs.client.hedged.read.threshold.millis</name>
<value>20000</value>
</property>
</configuration>
```
threshold is a really large value
was:
I configure hedged read about below config:
```xml
<configuration>
<property>
<name>dfs.client.hedged.read.threadpool.size</name>
<value>128</value>
</property>
<property>
<name>dfs.client.hedged.read.threshold.millis</name>
<value>2000</value>
</property>
</configuration>
```
> Hdfs client will easily to oom when enable hedged read
> ------------------------------------------------------
>
> Key: HADOOP-18835
> URL: https://issues.apache.org/jira/browse/HADOOP-18835
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 3.3.3
> Environment: I configure hedged read about below config:
> ```xml
> <configuration>
> <property>
> <name>dfs.client.hedged.read.threadpool.size</name>
> <value>128</value>
> </property>
> <property>
> <name>dfs.client.hedged.read.threshold.millis</name>
> <value>20000</value>
> </property>
> </configuration>
> ```
> threshold is a really large value
> Reporter: Smith Cruise
> Priority: Major
>
> In the same workload, when I disable hedged read, JVM heap is:
> ```bash
> Heap Configuration:
> MinHeapFreeRatio = 40
> MaxHeapFreeRatio = 70
> MaxHeapSize = 32178700288 (30688.0MB)
> NewSize = 1363144 (1.2999954223632812MB)
> MaxNewSize = 19306381312 (18412.0MB)
> OldSize = 5452592 (5.1999969482421875MB)
> NewRatio = 2
> SurvivorRatio = 8
> MetaspaceSize = 21807104 (20.796875MB)
> CompressedClassSpaceSize = 1073741824 (1024.0MB)
> MaxMetaspaceSize = 17592186044415 MB
> G1HeapRegionSize = 4194304 (4.0MB)
> Heap Usage:
> G1 Heap:
> regions = 7672
> capacity = 32178700288 (30688.0MB)
> used = 794118192 (757.3301239013672MB)
> free = 31384582096 (29930.669876098633MB)
> 2.467837994986207% used
> G1 Young Generation:
> Eden Space:
> regions = 177
> capacity = 1732247552 (1652.0MB)
> used = 742391808 (708.0MB)
> free = 989855744 (944.0MB)
> 42.857142857142854% used
> Survivor Space:
> regions = 6
> capacity = 25165824 (24.0MB)
> used = 25165824 (24.0MB)
> free = 0 (0.0MB)
> 100.0% used
> G1 Old Generation:
> regions = 7
> capacity = 1035993088 (988.0MB)
> used = 26560560 (25.330123901367188MB)
> free = 1009432528 (962.6698760986328MB)
> 2.563777722810444% used
> ```
>
> When I enable hedged read, it easily oom:
> ```bash
> preadDirect: FSDataInputStream#read error:
> OutOfMemoryError: Java heap spacejava.lang.OutOfMemoryError: Java heap space
> preadDirect: FSDataInputStream#read error:
> OutOfMemoryError: Java heap spacejava.lang.OutOfMemoryError: Java heap space
> at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:61)
> at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:348)
> at
> org.apache.hadoop.hdfs.DFSInputStream.hedgedFetchBlockByteRange(DFSInputStream.java:1292)
> at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1493)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1705)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:259)
> ```
>
> ```bash
> Heap Configuration:
> MinHeapFreeRatio = 40
> MaxHeapFreeRatio = 70
> MaxHeapSize = 32178700288 (30688.0MB)
> NewSize = 1363144 (1.2999954223632812MB)
> MaxNewSize = 19306381312 (18412.0MB)
> OldSize = 5452592 (5.1999969482421875MB)
> NewRatio = 2
> SurvivorRatio = 8
> MetaspaceSize = 21807104 (20.796875MB)
> CompressedClassSpaceSize = 1073741824 (1024.0MB)
> MaxMetaspaceSize = 17592186044415 MB
> G1HeapRegionSize = 4194304 (4.0MB)
> Heap Usage:
> G1 Heap:
> regions = 7672
> capacity = 32178700288 (30688.0MB)
> used = 14680397040 (14000.317611694336MB)
> free = 17498303248 (16687.682388305664MB)
> 45.62147292653264% used
> G1 Young Generation:
> Eden Space:
> regions = 1
> capacity = 11991515136 (11436.0MB)
> used = 4194304 (4.0MB)
> free = 11987320832 (11432.0MB)
> 0.03497726477789437% used
> Survivor Space:
> regions = 1
> capacity = 4194304 (4.0MB)
> used = 4194304 (4.0MB)
> free = 0 (0.0MB)
> 100.0% used
> G1 Old Generation:
> regions = 3500
> capacity = 20182990848 (19248.0MB)
> used = 14672008432 (13992.317611694336MB)
> free = 5510982416 (5255.682388305664MB)
> 72.69491693523658% used
> ```
>
> Any idea about this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
