liying created HDFS-14738:
-----------------------------
Summary: Reading and writing HDFS data through HDFS nfs3 server
often getting stuck
Key: HDFS-14738
URL: https://issues.apache.org/jira/browse/HDFS-14738
Project: Hadoop HDFS
Issue Type: Improvement
Components: nfs
Affects Versions: 2.7.2
Reporter: liying
Assignee: liying
We know the NFS Gateway supports NFSv3 and allows HDFS to be mounted as part
of the client’s local file system.
I start the portmap and nfs3server in the hadoop node，and use the command
（mount -t nfs -o vers=3,proto=tcp,nolock nfs3serverIP:/ /hdfs） to mount.
It works well。Then i ues this command to mount on many client.
And i used linux cp to copy files to the mounted dir(/hdfs), found the speed
was very slow . There is a lot of information in the log as the follow:
2019-08-14 15:36:28,347 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:03,093 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:03,850 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:14,780 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:14,928 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:28,410 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:09,310 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:10,069 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:14,856 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:14,957 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:28,475 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:14,923 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:14,987 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:15,541 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:16,287 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:28,530 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:15,015 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:15,024 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:15,028 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:21,757 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:22,508 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:28,583 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:15,063 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:15,086 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:27,983 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:28,647 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:28,729 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:42:15,184 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:42:15,191 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:42:28,707 INFO org.apache.hadoop.security.ShellBasedIdMapping:
Can't map group supergroup. Use its string hashcode:-1710818332
Then i print the jstack log of nfs3 server jvm, and i found that many threads
waiting for monitor entry : BLOCKED(on object monitor). After checking the
codes of ShellBasedIdMapping, found that It will invoke the updateMapInternal
every time if the user or group doesn't exits. And many methods in this class
are made with synchronized. So i think we can avoid to invoke updateMapInternal
every time.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
