[jira] [Updated] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[ https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-15039: --- Fix Version/s: (was: 3.4.0) 3.3.0 > Cache meta file length of FinalizedReplica to reduce call File.length() > --- > > Key: HDFS-15039 > URL: https://issues.apache.org/jira/browse/HDFS-15039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Fix For: 3.3.0 > > Attachments: HDFS-15039.006.patch, HDFS-15039.patch, > HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch > > > When use ReplicaCachingGetSpaceUsed to get the volume space used. It will > call File.length() for every meta file of replica. That add more disk IO, we > found the slow log as below. For finalized replica, the size of meta file is > not changed, i think we can cache the value. > {code:java} > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: > Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas > size: 1166 dfsUsed: 72227113183 on volume: > DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[ https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-15039: --- Fix Version/s: 3.4.0 > Cache meta file length of FinalizedReplica to reduce call File.length() > --- > > Key: HDFS-15039 > URL: https://issues.apache.org/jira/browse/HDFS-15039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15039.006.patch, HDFS-15039.patch, > HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch > > > When use ReplicaCachingGetSpaceUsed to get the volume space used. It will > call File.length() for every meta file of replica. That add more disk IO, we > found the slow log as below. For finalized replica, the size of meta file is > not changed, i think we can cache the value. > {code:java} > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: > Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas > size: 1166 dfsUsed: 72227113183 on volume: > DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[ https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15039: Resolution: Fixed Status: Resolved (was: Patch Available) > Cache meta file length of FinalizedReplica to reduce call File.length() > --- > > Key: HDFS-15039 > URL: https://issues.apache.org/jira/browse/HDFS-15039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15039.006.patch, HDFS-15039.patch, > HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch > > > When use ReplicaCachingGetSpaceUsed to get the volume space used. It will > call File.length() for every meta file of replica. That add more disk IO, we > found the slow log as below. For finalized replica, the size of meta file is > not changed, i think we can cache the value. > {code:java} > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: > Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas > size: 1166 dfsUsed: 72227113183 on volume: > DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[ https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-15039: --- Attachment: HDFS-15039.006.patch > Cache meta file length of FinalizedReplica to reduce call File.length() > --- > > Key: HDFS-15039 > URL: https://issues.apache.org/jira/browse/HDFS-15039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15039.006.patch, HDFS-15039.patch, > HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch > > > When use ReplicaCachingGetSpaceUsed to get the volume space used. It will > call File.length() for every meta file of replica. That add more disk IO, we > found the slow log as below. For finalized replica, the size of meta file is > not changed, i think we can cache the value. > {code:java} > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: > Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas > size: 1166 dfsUsed: 72227113183 on volume: > DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[ https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15039: Attachment: HDFS-15039.patch Status: Patch Available (was: Open) > Cache meta file length of FinalizedReplica to reduce call File.length() > --- > > Key: HDFS-15039 > URL: https://issues.apache.org/jira/browse/HDFS-15039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch, > HDFS-15039.patch, HDFS-15039.patch > > > When use ReplicaCachingGetSpaceUsed to get the volume space used. It will > call File.length() for every meta file of replica. That add more disk IO, we > found the slow log as below. For finalized replica, the size of meta file is > not changed, i think we can cache the value. > {code:java} > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: > Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas > size: 1166 dfsUsed: 72227113183 on volume: > DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[ https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15039: Status: Open (was: Patch Available) > Cache meta file length of FinalizedReplica to reduce call File.length() > --- > > Key: HDFS-15039 > URL: https://issues.apache.org/jira/browse/HDFS-15039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch, > HDFS-15039.patch, HDFS-15039.patch > > > When use ReplicaCachingGetSpaceUsed to get the volume space used. It will > call File.length() for every meta file of replica. That add more disk IO, we > found the slow log as below. For finalized replica, the size of meta file is > not changed, i think we can cache the value. > {code:java} > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: > Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas > size: 1166 dfsUsed: 72227113183 on volume: > DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[ https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15039: Attachment: HDFS-15039.patch Status: Patch Available (was: Open) > Cache meta file length of FinalizedReplica to reduce call File.length() > --- > > Key: HDFS-15039 > URL: https://issues.apache.org/jira/browse/HDFS-15039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch, > HDFS-15039.patch > > > When use ReplicaCachingGetSpaceUsed to get the volume space used. It will > call File.length() for every meta file of replica. That add more disk IO, we > found the slow log as below. For finalized replica, the size of meta file is > not changed, i think we can cache the value. > {code:java} > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: > Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas > size: 1166 dfsUsed: 72227113183 on volume: > DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[ https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15039: Status: Open (was: Patch Available) > Cache meta file length of FinalizedReplica to reduce call File.length() > --- > > Key: HDFS-15039 > URL: https://issues.apache.org/jira/browse/HDFS-15039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch > > > When use ReplicaCachingGetSpaceUsed to get the volume space used. It will > call File.length() for every meta file of replica. That add more disk IO, we > found the slow log as below. For finalized replica, the size of meta file is > not changed, i think we can cache the value. > {code:java} > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: > Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas > size: 1166 dfsUsed: 72227113183 on volume: > DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[ https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15039: Attachment: HDFS-15039.patch Status: Patch Available (was: Open) > Cache meta file length of FinalizedReplica to reduce call File.length() > --- > > Key: HDFS-15039 > URL: https://issues.apache.org/jira/browse/HDFS-15039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15039.patch, HDFS-15039.patch, HDFS-15039.patch > > > When use ReplicaCachingGetSpaceUsed to get the volume space used. It will > call File.length() for every meta file of replica. That add more disk IO, we > found the slow log as below. For finalized replica, the size of meta file is > not changed, i think we can cache the value. > {code:java} > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: > Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas > size: 1166 dfsUsed: 72227113183 on volume: > DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[ https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15039: Attachment: HDFS-15039.patch > Cache meta file length of FinalizedReplica to reduce call File.length() > --- > > Key: HDFS-15039 > URL: https://issues.apache.org/jira/browse/HDFS-15039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15039.patch, HDFS-15039.patch > > > When use ReplicaCachingGetSpaceUsed to get the volume space used. It will > call File.length() for every meta file of replica. That add more disk IO, we > found the slow log as below. For finalized replica, the size of meta file is > not changed, i think we can cache the value. > {code:java} > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: > Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas > size: 1166 dfsUsed: 72227113183 on volume: > DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[ https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15039: Status: Open (was: Patch Available) > Cache meta file length of FinalizedReplica to reduce call File.length() > --- > > Key: HDFS-15039 > URL: https://issues.apache.org/jira/browse/HDFS-15039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15039.patch > > > When use ReplicaCachingGetSpaceUsed to get the volume space used. It will > call File.length() for every meta file of replica. That add more disk IO, we > found the slow log as below. For finalized replica, the size of meta file is > not changed, i think we can cache the value. > {code:java} > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: > Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas > size: 1166 dfsUsed: 72227113183 on volume: > DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15039) Cache meta file length of FinalizedReplica to reduce call File.length()
[ https://issues.apache.org/jira/browse/HDFS-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15039: Attachment: HDFS-15039.patch Status: Patch Available (was: Open) > Cache meta file length of FinalizedReplica to reduce call File.length() > --- > > Key: HDFS-15039 > URL: https://issues.apache.org/jira/browse/HDFS-15039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15039.patch > > > When use ReplicaCachingGetSpaceUsed to get the volume space used. It will > call File.length() for every meta file of replica. That add more disk IO, we > found the slow log as below. For finalized replica, the size of meta file is > not changed, i think we can cache the value. > {code:java} > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaCachingGetSpaceUsed: > Refresh dfs used, bpid: BP-898717543-10.75.1.240-1519386995727 replicas > size: 1166 dfsUsed: 72227113183 on volume: > DS-3add8d62-d69a-4f5a-a29f-b7bbb400af2e duration: 17206ms{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org