[jira] [Commented] (HDFS-9260) Improve the performance and GC friendliness of NameNode startup and full block reports
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558382#comment-16558382 ] Kihwal Lee commented on HDFS-9260: -- I propose revert of this. HDFS-13671 also reports about 4x slower performance. It might help GC, but regular operations are being affected too much. > Improve the performance and GC friendliness of NameNode startup and full > block reports > -- > > Key: HDFS-9260 > URL: https://issues.apache.org/jira/browse/HDFS-9260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg >Priority: Major > Fix For: 3.0.0-alpha1 > > Attachments: FBR processing.png, HDFS Block and Replica Management > 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch, > HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch, > HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch, > HDFS-9260.010.patch, HDFS-9260.011.patch, HDFS-9260.012.patch, > HDFS-9260.013.patch, HDFS-9260.014.patch, HDFS-9260.015.patch, > HDFS-9260.016.patch, HDFS-9260.017.patch, HDFS-9260.018.patch, > HDFSBenchmarks.zip, HDFSBenchmarks2.zip > > > This patch changes the datastructures used for BlockInfos and Replicas to > keep them sorted. This allows faster and more GC friendly handling of full > block reports. > Would like to hear peoples feedback on this change. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9260) Improve the performance and GC friendliness of NameNode startup and full block reports
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966680#comment-15966680 ] Andrew Wang commented on HDFS-9260: --- One other question from our team, what's the typical block size on a Yahoo cluster? > Improve the performance and GC friendliness of NameNode startup and full > block reports > -- > > Key: HDFS-9260 > URL: https://issues.apache.org/jira/browse/HDFS-9260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg > Fix For: 3.0.0-alpha1 > > Attachments: FBR processing.png, HDFS-7435.001.patch, > HDFS-7435.002.patch, HDFS-7435.003.patch, HDFS-7435.004.patch, > HDFS-7435.005.patch, HDFS-7435.006.patch, HDFS-7435.007.patch, > HDFS-9260.008.patch, HDFS-9260.009.patch, HDFS-9260.010.patch, > HDFS-9260.011.patch, HDFS-9260.012.patch, HDFS-9260.013.patch, > HDFS-9260.014.patch, HDFS-9260.015.patch, HDFS-9260.016.patch, > HDFS-9260.017.patch, HDFS-9260.018.patch, HDFSBenchmarks2.zip, > HDFSBenchmarks.zip, HDFS Block and Replica Management 20151013.pdf > > > This patch changes the datastructures used for BlockInfos and Replicas to > keep them sorted. This allows faster and more GC friendly handling of full > block reports. > Would like to hear peoples feedback on this change. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9260) Improve the performance and GC friendliness of NameNode startup and full block reports
[
https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966423#comment-15966423
]
Andrew Wang commented on HDFS-9260:
---
Hi Daryn,
I talked to our people who help run this large customer cluster. It's at about
350 million blocks, so a pretty good size, but also a lot denser than the last
published stats I saw about the 4500-node Yahoo cluster. We don't have
historical GC metrics going back a year when we put this into CDH, but they
haven't seen anything abnormal in terms of GC.
They were quite interested your balancer settings though, since we haven't seen
it stressing the NN. Could you provide the following?
{noformat}
dfs.datanode.balance.bandwidthPerSec
dfs.datanode.balance.max.concurrent.moves
dfs.namenode.replication.work.multiplier.per.iteration
dfs.namenode.replication.max-streams-hard-limit
{noformat}
I believe we're running it with mostly default settings like this:
{noformat}
hdfs balancer -Ddfs.datanode.balance.max.concurrent.moves=200 -threshold 10
{noformat}
> Improve the performance and GC friendliness of NameNode startup and full
> block reports
> --
>
> Key: HDFS-9260
> URL: https://issues.apache.org/jira/browse/HDFS-9260
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, performance
>Affects Versions: 2.7.1
>Reporter: Staffan Friberg
>Assignee: Staffan Friberg
> Fix For: 3.0.0-alpha1
>
> Attachments: FBR processing.png, HDFS-7435.001.patch,
> HDFS-7435.002.patch, HDFS-7435.003.patch, HDFS-7435.004.patch,
> HDFS-7435.005.patch, HDFS-7435.006.patch, HDFS-7435.007.patch,
> HDFS-9260.008.patch, HDFS-9260.009.patch, HDFS-9260.010.patch,
> HDFS-9260.011.patch, HDFS-9260.012.patch, HDFS-9260.013.patch,
> HDFS-9260.014.patch, HDFS-9260.015.patch, HDFS-9260.016.patch,
> HDFS-9260.017.patch, HDFS-9260.018.patch, HDFSBenchmarks2.zip,
> HDFSBenchmarks.zip, HDFS Block and Replica Management 20151013.pdf
>
>
> This patch changes the datastructures used for BlockInfos and Replicas to
> keep them sorted. This allows faster and more GC friendly handling of full
> block reports.
> Would like to hear peoples feedback on this change.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9260) Improve the performance and GC friendliness of NameNode startup and full block reports
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951459#comment-15951459 ] Daryn Sharp commented on HDFS-9260: --- I have no doubt this patch "works". If you are collecting GC metrics check if you see elevation or spikes in frequency or cpu over historic levels. Roughly what are the specs for the "highest scale" you have observed? This blocks HDFS-7967 which is a critical feature for our clusters - It's been a deployment blocker since 2.6. The balancer is unusable with 500 million to over 1 billion replicas on dense storages. I have detailed in a comment on HDFS-7967 that getBlocks will take hundreds of ms. Sustaining 20-40k average ops/sec is impossible. Rolling upgrades also already place very high GC pressure on the NN, some narrowly escaping a full GC. I'm very worried if the NN has to re-sort FBRs during the upgrade. I'd like to stop internally maintaining HDFS-7967 and let the community benefit. May we please revert? > Improve the performance and GC friendliness of NameNode startup and full > block reports > -- > > Key: HDFS-9260 > URL: https://issues.apache.org/jira/browse/HDFS-9260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg > Fix For: 3.0.0-alpha1 > > Attachments: FBR processing.png, HDFS-7435.001.patch, > HDFS-7435.002.patch, HDFS-7435.003.patch, HDFS-7435.004.patch, > HDFS-7435.005.patch, HDFS-7435.006.patch, HDFS-7435.007.patch, > HDFS-9260.008.patch, HDFS-9260.009.patch, HDFS-9260.010.patch, > HDFS-9260.011.patch, HDFS-9260.012.patch, HDFS-9260.013.patch, > HDFS-9260.014.patch, HDFS-9260.015.patch, HDFS-9260.016.patch, > HDFS-9260.017.patch, HDFS-9260.018.patch, HDFSBenchmarks2.zip, > HDFSBenchmarks.zip, HDFS Block and Replica Management 20151013.pdf > > > This patch changes the datastructures used for BlockInfos and Replicas to > keep them sorted. This allows faster and more GC friendly handling of full > block reports. > Would like to hear peoples feedback on this change. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9260) Improve the performance and GC friendliness of NameNode startup and full block reports
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819520#comment-15819520 ] Andrew Wang commented on HDFS-9260: --- FWIW this has been in a released version of CDH for 10 months, and is running in production at some of our highest scale customers. We did see one issue where we mistakenly weren't sending sorted FBRs even after an upgrade, but nothing since this was fixed. Daryn, what metrics are you looking for specifically? I can ask our field about collecting these on some big clusters. > Improve the performance and GC friendliness of NameNode startup and full > block reports > -- > > Key: HDFS-9260 > URL: https://issues.apache.org/jira/browse/HDFS-9260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg > Fix For: 3.0.0-alpha1 > > Attachments: FBR processing.png, HDFS Block and Replica Management > 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch, > HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch, > HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch, > HDFS-9260.010.patch, HDFS-9260.011.patch, HDFS-9260.012.patch, > HDFS-9260.013.patch, HDFS-9260.014.patch, HDFS-9260.015.patch, > HDFS-9260.016.patch, HDFS-9260.017.patch, HDFS-9260.018.patch, > HDFSBenchmarks.zip, HDFSBenchmarks2.zip > > > This patch changes the datastructures used for BlockInfos and Replicas to > keep them sorted. This allows faster and more GC friendly handling of full > block reports. > Would like to hear peoples feedback on this change. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9260) Improve the performance and GC friendliness of NameNode startup and full block reports
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815889#comment-15815889 ] Daryn Sharp commented on HDFS-9260: --- I'd like to propose this patch be reverted on the NN, but perhaps not the DN. Partially because it complicates porting HDFS-11310, but also due to performance concerns. Synopsis of the design change: Reference-dense datastructures with high mutation rates are the enemy of young gen gc pauses. When a tenured reference is updated, a dirty card tracks the memory region of the changed reference. The next young gen gc checks traceability to references in the dirty card regions. FBR processing rewrites triplets references to identity unreported blocks, causing a spike in young gen gc load. A folded/sorted tree is a clever means to avoid rewriting the triplets pointer during a FBR, thus reducing young gen gc pressure. That said, here's the main issues I see: # FBRs: DNs must send sorted block reports, else the NN creates a temporary sorted/folded tree for iteration. A new block report encoding was designed (by me) for in-place iteration to reduce high object allocation rates that made FBR processing unacceptable at scale. Building the sorted tree undoes the benefit by exploding the entire report into a highly fragmented tree (out of order insertion). During a rolling upgrade this will place extreme pressure on the NN until all DNs are upgraded. # IBRs: The performance of the tree is predicated on increasing block ids to avoid fragmentation while filling the tree. However, block churn and organic replication from dead nodes, decommissioning nodes, failed storages, block balancing, etc will pepper a node with random blocks. The IBRs will cause trees to quickly fragment. The tree mutations are likely to cause more dirty cards than simply linking/unlinking a block into the triplets. # Tree compaction: Every 10 mins all sufficiently fragmented storage trees will be compacted. In practice this may be a large portion of the cluster storages due to IBRs, translating to bursts of very heavy gc load. Heap growth will increase due to defunct tree nodes. # The CMS remark time reduction is not compelling when cycles should occur every few days or a week if the heap is adequately sized. The doc primarily focuses on FBRs with a footnote of 4X increase in IBR processing and negative impacts to balancing. Impacts to balancing are equivalent to replication from dead nodes, failed storages, decommissioning nodes, invalidations when dead nodes rejoin, etc. FBR savings are great, but not at the expense of increased processing and gc load from IBRs. Unless there are real-world metrics with a large cluster under load that dispute my concerns, I think we should revert. > Improve the performance and GC friendliness of NameNode startup and full > block reports > -- > > Key: HDFS-9260 > URL: https://issues.apache.org/jira/browse/HDFS-9260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg > Fix For: 3.0.0-alpha1 > > Attachments: FBR processing.png, HDFS Block and Replica Management > 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch, > HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch, > HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch, > HDFS-9260.010.patch, HDFS-9260.011.patch, HDFS-9260.012.patch, > HDFS-9260.013.patch, HDFS-9260.014.patch, HDFS-9260.015.patch, > HDFS-9260.016.patch, HDFS-9260.017.patch, HDFS-9260.018.patch, > HDFSBenchmarks.zip, HDFSBenchmarks2.zip > > > This patch changes the datastructures used for BlockInfos and Replicas to > keep them sorted. This allows faster and more GC friendly handling of full > block reports. > Would like to hear peoples feedback on this change. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9260) Improve the performance and GC friendliness of NameNode startup and full block reports
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197863#comment-15197863 ] Colin Patrick McCabe commented on HDFS-9260: Sure, would you like to backport it to 2.9? > Improve the performance and GC friendliness of NameNode startup and full > block reports > -- > > Key: HDFS-9260 > URL: https://issues.apache.org/jira/browse/HDFS-9260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg > Fix For: 3.0.0 > > Attachments: FBR processing.png, HDFS Block and Replica Management > 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch, > HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch, > HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch, > HDFS-9260.010.patch, HDFS-9260.011.patch, HDFS-9260.012.patch, > HDFS-9260.013.patch, HDFS-9260.014.patch, HDFS-9260.015.patch, > HDFS-9260.016.patch, HDFS-9260.017.patch, HDFS-9260.018.patch, > HDFSBenchmarks.zip, HDFSBenchmarks2.zip > > > This patch changes the datastructures used for BlockInfos and Replicas to > keep them sorted. This allows faster and more GC friendly handling of full > block reports. > Would like to hear peoples feedback on this change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9260) Improve the performance and GC friendliness of NameNode startup and full block reports
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196630#comment-15196630 ] Vinayakumar B commented on HDFS-9260: - How about bringing this into branch-2? > Improve the performance and GC friendliness of NameNode startup and full > block reports > -- > > Key: HDFS-9260 > URL: https://issues.apache.org/jira/browse/HDFS-9260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg > Fix For: 3.0.0 > > Attachments: FBR processing.png, HDFS Block and Replica Management > 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch, > HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch, > HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch, > HDFS-9260.010.patch, HDFS-9260.011.patch, HDFS-9260.012.patch, > HDFS-9260.013.patch, HDFS-9260.014.patch, HDFS-9260.015.patch, > HDFS-9260.016.patch, HDFS-9260.017.patch, HDFS-9260.018.patch, > HDFSBenchmarks.zip, HDFSBenchmarks2.zip > > > This patch changes the datastructures used for BlockInfos and Replicas to > keep them sorted. This allows faster and more GC friendly handling of full > block reports. > Would like to hear peoples feedback on this change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9260) Improve the performance and GC friendliness of NameNode startup and full block reports
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128869#comment-15128869 ] Hudson commented on HDFS-9260: -- FAILURE: Integrated in Hadoop-trunk-Commit #9227 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9227/]) HDFS-9260. Improve the performance and GC friendliness of NameNode (cmccabe: rev dd9ebf6eedfd4ff8b3486eae2a446de6b0c7fa8a) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockReportContext.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDnRespectsBlockReportSplitThreshold.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestNNHandlesCombinedBlockReport.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguous.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/FoldedTreeSet.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/FoldedTreeSetTest.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestNNHandlesBlockReportPerStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockHasMultipleReplicasOnSameDN.java > Improve the performance and GC friendliness of NameNode startup and full > block reports > -- > > Key: HDFS-9260 > URL: https://issues.apache.org/jira/browse/HDFS-9260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg > Attachments: FBR processing.png, HDFS Block and Replica Management > 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch, > HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch, > HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch, > HDFS-9260.010.patch, HDFS-9260.011.patch, HDFS-9260.012.patch, > HDFS-9260.013.patch, HDFS-9260.014.patch, HDFS-9260.015.patch, > HDFS-9260.016.patch, HDFS-9260.017.patch, HDFS-9260.018.patch, > HDFSBenchmarks.zip, HDFSBenchmarks2.zip > > > This patch changes the datastructures used for
[jira] [Commented] (HDFS-9260) Improve the performance and GC friendliness of NameNode startup and full block reports
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128986#comment-15128986 ] Hudson commented on HDFS-9260: -- FAILURE: Integrated in Hadoop-trunk-Commit #9230 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9230/]) CHANGES.txt: Move HDFS-9260 to trunk (cmccabe: rev 913676dc355f17dc41b75be1b3a27114197ea52c) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Improve the performance and GC friendliness of NameNode startup and full > block reports > -- > > Key: HDFS-9260 > URL: https://issues.apache.org/jira/browse/HDFS-9260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg > Fix For: 3.0.0 > > Attachments: FBR processing.png, HDFS Block and Replica Management > 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch, > HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch, > HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch, > HDFS-9260.010.patch, HDFS-9260.011.patch, HDFS-9260.012.patch, > HDFS-9260.013.patch, HDFS-9260.014.patch, HDFS-9260.015.patch, > HDFS-9260.016.patch, HDFS-9260.017.patch, HDFS-9260.018.patch, > HDFSBenchmarks.zip, HDFSBenchmarks2.zip > > > This patch changes the datastructures used for BlockInfos and Replicas to > keep them sorted. This allows faster and more GC friendly handling of full > block reports. > Would like to hear peoples feedback on this change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
