[
https://issues.apache.org/jira/browse/MAPREDUCE-7460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ECFuzz updated MAPREDUCE-7460:
------------------------------
Description:
My hadoop version is 3.3.6, and I use the Pseudo-Distributed Operation.
If yarn.nodemanager.resource.memory-mb and mapreduce.map.memory.mb violate the
relationship, it will cause the mapreduce sample program to block.
The first configuraion should be larger than the second one, but we manually
violate the relationship.
Here we provide the entire process.
core-site.xml like below.
{code:java}
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/lfl/Mutil_Component/tmp</value>
</property>
</configuration>{code}
hdfs-site.xml like below.
{noformat}
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
{noformat}
And then format the namenode, and start the hdfs. HDFS is running normally.
{noformat}
lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$ ./bin/hdfs namenode -format
lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$
./sbin/start-dfs.sh{noformat}
We add yarn.nodemanager.resource.memory-mb to yarn-site.xml like below.
{noformat}
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>1024</value>
</property>{noformat}
Also, we alse add mapreduce.map.memory.mb to mapred-site.xml like below.
{code:java}
<property>
<name>mapreduce.map.memory.mb</name>
<value>2048</value>
</property> {code}
Finally, we start the yarn and run the given mapreduce sample job.
{code:java}
lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$ bin/hdfs dfs -mkdir -p
/user/lfl
lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$ bin/hdfs dfs -mkdir input
lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$ bin/hdfs dfs -put
etc/hadoop/*.xml input
lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$ sbin/start-yarn.sh
Starting resourcemanager
Starting nodemanagers
lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input output
'dfs[a-z.]+'{code}
{code:java}
2023-10-26 10:44:47,027 INFO client.DefaultNoHARMFailoverProxyProvider:
Connecting to ResourceManager at localhost/127.0.0.1:8032
2023-10-26 10:44:47,288 INFO mapreduce.JobResourceUploader: Disabling Erasure
Coding for path: /tmp/hadoop-yarn/staging/lfl/.staging/job_1698288283355_0001
2023-10-26 10:44:47,483 INFO input.FileInputFormat: Total input files to
process : 10
2023-10-26 10:44:47,930 INFO mapreduce.JobSubmitter: number of splits:10
2023-10-26 10:44:48,406 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1698288283355_0001
2023-10-26 10:44:48,406 INFO mapreduce.JobSubmitter: Executing with tokens: []
2023-10-26 10:44:48,513 INFO conf.Configuration: resource-types.xml not found
2023-10-26 10:44:48,513 INFO resource.ResourceUtils: Unable to find
'resource-types.xml'.
2023-10-26 10:44:48,669 INFO impl.YarnClientImpl: Submitted application
application_1698288283355_0001
2023-10-26 10:44:48,703 INFO mapreduce.Job: The url to track the job:
http://LAPTOP-QR7GJ7B1.localdomain:8088/proxy/application_1698288283355_0001/
2023-10-26 10:44:48,703 INFO mapreduce.Job: Running job: job_1698288283355_0001
{code}
And, it will continue to be blocked here.
was:
My hadoop version is 3.3.6, and I use the Pseudo-Distributed Operation.
If configuration a and configuration b violate the constraint relationship, it
will cause the mapreduce sample program to block
core-site.xml like below.
{code:java}
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/lfl/Mutil_Component/tmp</value>
</property>
</configuration>{code}
hdfs-site.xml like below.
{noformat}
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
{noformat}
And then format the namenode, and start the hdfs. HDFS is running normally.
{noformat}
lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$ ./bin/hdfs namenode -format
lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$
./sbin/start-dfs.sh{noformat}
We add yarn.nodemanager.resource.memory-mb to yarn-site.xml like below.
{noformat}
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>1024</value>
</property>{noformat}
Also, we alse add mapreduce.map.memory.mb to mapred-site.xml like below.
{code:java}
<property>
<name>mapreduce.map.memory.mb</name>
<value>2048</value>
</property> {code}
> When "yarn.nodemanager.resource.memory-mb" and "mapreduce.map.memory.mb" work
> together, the mapreduce sample program blocks
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-7460
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7460
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: yarn
> Affects Versions: 3.3.6
> Reporter: ECFuzz
> Priority: Major
>
> My hadoop version is 3.3.6, and I use the Pseudo-Distributed Operation.
> If yarn.nodemanager.resource.memory-mb and mapreduce.map.memory.mb violate
> the relationship, it will cause the mapreduce sample program to block.
> The first configuraion should be larger than the second one, but we manually
> violate the relationship.
> Here we provide the entire process.
> core-site.xml like below.
> {code:java}
> <configuration>
> <property>
> <name>fs.defaultFS</name>
> <value>hdfs://localhost:9000</value>
> </property>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/home/lfl/Mutil_Component/tmp</value>
> </property>
>
> </configuration>{code}
> hdfs-site.xml like below.
> {noformat}
> <configuration>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
> </configuration>
> {noformat}
> And then format the namenode, and start the hdfs. HDFS is running normally.
> {noformat}
> lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$ ./bin/hdfs namenode
> -format
> lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$
> ./sbin/start-dfs.sh{noformat}
>
> We add yarn.nodemanager.resource.memory-mb to yarn-site.xml like below.
> {noformat}
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>1024</value>
> </property>{noformat}
> Also, we alse add mapreduce.map.memory.mb to mapred-site.xml like below.
> {code:java}
> <property>
> <name>mapreduce.map.memory.mb</name>
> <value>2048</value>
> </property> {code}
>
> Finally, we start the yarn and run the given mapreduce sample job.
>
> {code:java}
> lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$ bin/hdfs dfs -mkdir -p
> /user/lfl
> lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$ bin/hdfs dfs -mkdir input
> lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$ bin/hdfs dfs -put
> etc/hadoop/*.xml input
> lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$ sbin/start-yarn.sh
> Starting resourcemanager
> Starting nodemanagers
> lfl@LAPTOP-QR7GJ7B1:~/Mutil_Component/hadoop-3.3.6$bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input output
> 'dfs[a-z.]+'{code}
> {code:java}
> 2023-10-26 10:44:47,027 INFO client.DefaultNoHARMFailoverProxyProvider:
> Connecting to ResourceManager at localhost/127.0.0.1:8032
> 2023-10-26 10:44:47,288 INFO mapreduce.JobResourceUploader: Disabling Erasure
> Coding for path: /tmp/hadoop-yarn/staging/lfl/.staging/job_1698288283355_0001
> 2023-10-26 10:44:47,483 INFO input.FileInputFormat: Total input files to
> process : 10
> 2023-10-26 10:44:47,930 INFO mapreduce.JobSubmitter: number of splits:10
> 2023-10-26 10:44:48,406 INFO mapreduce.JobSubmitter: Submitting tokens for
> job: job_1698288283355_0001
> 2023-10-26 10:44:48,406 INFO mapreduce.JobSubmitter: Executing with tokens: []
> 2023-10-26 10:44:48,513 INFO conf.Configuration: resource-types.xml not found
> 2023-10-26 10:44:48,513 INFO resource.ResourceUtils: Unable to find
> 'resource-types.xml'.
> 2023-10-26 10:44:48,669 INFO impl.YarnClientImpl: Submitted application
> application_1698288283355_0001
> 2023-10-26 10:44:48,703 INFO mapreduce.Job: The url to track the job:
> http://LAPTOP-QR7GJ7B1.localdomain:8088/proxy/application_1698288283355_0001/
> 2023-10-26 10:44:48,703 INFO mapreduce.Job: Running job:
> job_1698288283355_0001 {code}
> And, it will continue to be blocked here.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
