[jira] [Updated] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6914: - Labels: hdfs (was: BB2015-05-TBR hdfs) > Resolve huge memory consumption Issue with OIV processing PB-based fsimages > --- > > Key: HDFS-6914 > URL: https://issues.apache.org/jira/browse/HDFS-6914 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Hao Chen > Labels: hdfs > Attachments: HDFS-6914.patch, HDFS-6914.v2.patch > > > For better managing and supporting a lot of large hadoop clusters in > production, we internally need to automatically export fsimage to delimited > text files in LSR style and then analyse with hive or pig or build system > metrics for real-time analyzing. > However due to the internal layout changes introduced by the protobuf-based > fsimage, OIV processing program consumes excessive amount of memory. For > example, in order to export the fsimage in size of 8GB, it should have taken > about 85GB memory which is really not reasonable and impacted performance of > other services badly in the same server. > To resolve above problem, I submit this patch which will reduce memory > consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6914: - Target Version/s: (was: 2.8.0) > Resolve huge memory consumption Issue with OIV processing PB-based fsimages > --- > > Key: HDFS-6914 > URL: https://issues.apache.org/jira/browse/HDFS-6914 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Hao Chen > Labels: hdfs > Attachments: HDFS-6914.patch, HDFS-6914.v2.patch > > > For better managing and supporting a lot of large hadoop clusters in > production, we internally need to automatically export fsimage to delimited > text files in LSR style and then analyse with hive or pig or build system > metrics for real-time analyzing. > However due to the internal layout changes introduced by the protobuf-based > fsimage, OIV processing program consumes excessive amount of memory. For > example, in order to export the fsimage in size of 8GB, it should have taken > about 85GB memory which is really not reasonable and impacted performance of > other services badly in the same server. > To resolve above problem, I submit this patch which will reduce memory > consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-6914: --- Labels: BB2015-05-TBR hdfs (was: hdfs) Resolve huge memory consumption Issue with OIV processing PB-based fsimages --- Key: HDFS-6914 URL: https://issues.apache.org/jira/browse/HDFS-6914 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.1 Reporter: Hao Chen Labels: BB2015-05-TBR, hdfs Attachments: HDFS-6914.patch, HDFS-6914.v2.patch For better managing and supporting a lot of large hadoop clusters in production, we internally need to automatically export fsimage to delimited text files in LSR style and then analyse with hive or pig or build system metrics for real-time analyzing. However due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server. To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Chen updated HDFS-6914: --- Fix Version/s: 2.4.1 Resolve huge memory consumption Issue with OIV processing PB-based fsimages --- Key: HDFS-6914 URL: https://issues.apache.org/jira/browse/HDFS-6914 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.1 Reporter: Hao Chen Labels: hdfs Fix For: 2.4.1 Attachments: HDFS-6914.patch For better managing and supporting a lot of large hadoop clusters in production, we internally need to automatically export fsimage to delimited text files in LSR style and then analyse with hive or pig or build system metrics for real-time analyzing. However due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server. To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Chen updated HDFS-6914: --- Component/s: hdfs-client Resolve huge memory consumption Issue with OIV processing PB-based fsimages --- Key: HDFS-6914 URL: https://issues.apache.org/jira/browse/HDFS-6914 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Affects Versions: 2.4.1 Reporter: Hao Chen Labels: hdfs Fix For: 2.4.1 Attachments: HDFS-6914.patch For better managing and supporting a lot of large hadoop clusters in production, we internally need to automatically export fsimage to delimited text files in LSR style and then analyse with hive or pig or build system metrics for real-time analyzing. However due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server. To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Chen updated HDFS-6914: --- Fix Version/s: (was: 2.4.1) Resolve huge memory consumption Issue with OIV processing PB-based fsimages --- Key: HDFS-6914 URL: https://issues.apache.org/jira/browse/HDFS-6914 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Affects Versions: 2.4.1 Reporter: Hao Chen Labels: hdfs Attachments: HDFS-6914.patch For better managing and supporting a lot of large hadoop clusters in production, we internally need to automatically export fsimage to delimited text files in LSR style and then analyse with hive or pig or build system metrics for real-time analyzing. However due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server. To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Chen updated HDFS-6914: --- Attachment: HDFS-6914.v2.patch Clean code to keep changes minimal and make the coding style consistent Resolve huge memory consumption Issue with OIV processing PB-based fsimages --- Key: HDFS-6914 URL: https://issues.apache.org/jira/browse/HDFS-6914 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Affects Versions: 2.4.1 Reporter: Hao Chen Labels: hdfs Attachments: HDFS-6914.patch, HDFS-6914.v2.patch For better managing and supporting a lot of large hadoop clusters in production, we internally need to automatically export fsimage to delimited text files in LSR style and then analyse with hive or pig or build system metrics for real-time analyzing. However due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server. To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Chen updated HDFS-6914: --- Component/s: (was: hdfs-client) Resolve huge memory consumption Issue with OIV processing PB-based fsimages --- Key: HDFS-6914 URL: https://issues.apache.org/jira/browse/HDFS-6914 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.1 Reporter: Hao Chen Labels: hdfs Attachments: HDFS-6914.patch, HDFS-6914.v2.patch For better managing and supporting a lot of large hadoop clusters in production, we internally need to automatically export fsimage to delimited text files in LSR style and then analyse with hive or pig or build system metrics for real-time analyzing. However due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server. To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HDFS-6914: --- Target Version/s: 2.6.0 (was: 2.5.1) Fix Version/s: (was: 2.5.1) Targeting 2.6.0. Also, the fixVersion is to be set at commit time by the person committing. Resolve huge memory consumption Issue with OIV processing PB-based fsimages --- Key: HDFS-6914 URL: https://issues.apache.org/jira/browse/HDFS-6914 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.1 Reporter: Hao Chen Labels: hdfs Attachments: HDFS-6914.patch For better managing and supporting a lot of large hadoop clusters in production, we internally need to automatically export fsimage to delimited text files in LSR style and then analyse with hive or pig or build system metrics for real-time analyzing. However due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server. To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Chen updated HDFS-6914: --- Attachment: HDFS-6914.patch Resolve huge memory consumption Issue with OIV processing PB-based fsimages --- Key: HDFS-6914 URL: https://issues.apache.org/jira/browse/HDFS-6914 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.1 Reporter: Hao Chen Labels: hdfs Fix For: 2.5.1 Attachments: HDFS-6914.patch For better managing and supporting a lot of large hadoop clusters in production, we internally need to automatically export fsimage to delimited text files in LSR style and then analyse with hive or pig or build system metrics for real-time analyzing. However due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server. To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Chen updated HDFS-6914: --- Status: Patch Available (was: Open) Due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server. To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by about half. And the solution is very simple. Instead of holding the whole deserialized INode object in memory, this patch just holds the attribute `name` of INode as node of the tree structure in heap. Resolve huge memory consumption Issue with OIV processing PB-based fsimages --- Key: HDFS-6914 URL: https://issues.apache.org/jira/browse/HDFS-6914 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.1 Reporter: Hao Chen Labels: hdfs Fix For: 2.5.1 Attachments: HDFS-6914.patch For better managing and supporting a lot of large hadoop clusters in production, we internally need to automatically export fsimage to delimited text files in LSR style and then analyse with hive or pig or build system metrics for real-time analyzing. However due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server. To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.2#6252)
