[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[
https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562108#comment-14562108
]
Allen Wittenauer commented on HDFS-6261:
Awesome! Doc patches are great!
Now for the review. ;) This isn't comprehensive, but here's a first pass at
least.
A common problem is a missing 'the' in front of 'following'. I pointed it out
in a few places, but there are more. Articles in English are tricky. :(
'Following' is particularly tricky though because 'following' without 'the' or
'a' in front of it is a verb (e.g., The dog was following the boy vs. Hadoop
has a following or See the following list of cool places to eat.). But hey,
at least we don't have genders like the European languages though! :)
{code}
However, for
+other cases, like: Hadoop nodes running on virtualized platform, we have
+additional hypervisor layer, and its characteristics include:
{code}
I don't know how to parse this phrasing. It feels awkward. I'd probably
rewrite as:
However for some cases, this is insufficient. Take for example Hadoop nodes
running on a virtualized platform where there is an additional hypervisor
layer. It has the following characteristics:
{code}
+- The communication price between VMs within the same hypervisor is lower
+than across hypervisor (physical host) which will have higher throughput,
+lower latency, and not generating physical network traffic.
{code}
Same sort of problem. I'd probably rephrase a bit:
The communication price between multiple VMs running on one physical host is
lower than the communication price between processes on multiple physical
hosts. In addition to the multiple VMs having higher throughput and lower
latency between themselves, they do not generate any network traffic on the
wire.
{code}
transparent for Hadoop, so
{code}
'for' should be 'to'. Hadoop (period). (new sentence) So
{code}
like following:
{code}
like the following:
{code}
layer, following polices
+in hdfs are refined:
{code}
the following. HDFS.
{code}
+- Replica placement policy
{code}
I have a feeling bullet points in front of all the items listed under this
section may render better. I need to play with it though.
{code}
of writer,
{code
of the writer
{code}
on other
+node
{code}
on another node
{code}
if node of writer
{code}
if the node of the writer
{code}
The remaining replicas are placed randomly across rack and node group to
+meet minimum restriction.
{code}
I'm confused by this since there are missing articles and/or plurals here.
Does this mean randomly across the remaining racks or randomly across all racks
including the writer's rack?
{code}
At node level
{code}
At the node level
{code}
At block level
{code}
At the block level
{code}
Reliability: By never placing more than one replicas on the same node
+group(physical host), in case of node group failure, only one replica is
+lost at maximum.
{code}
Awkward phrasing. I'd probably rewrite as:
Reliability: By never placing more than one replica in the same node
group (aka physical host), only one replica is lost at maximum in case of node
group failure.
{code}
rather than remote node
{code}
than a remote
{code}
+3-layer topology tends to support different failure and locality topologies
+which is primarily driven from the perspective of virtualization, however,
+it is also possible to use the feature support other scenarios, such as
+those relating to failures of power supplies, arbitrary sets of physical
+servers, or collections of servers from same hardware purchase cycle.
{code}
This paragraph feels like it should be up closer to the top of these changes.
Add document for enabling node group layer in HDFS
--
Key: HDFS-6261
URL: https://issues.apache.org/jira/browse/HDFS-6261
Project: Hadoop HDFS
Issue Type: Task
Components: documentation
Reporter: Wenwu Peng
Assignee: Binglin Chang
Labels: documentation
Attachments: 2-layer-topology.png, 3-layer-topology.png,
3layer-topology.png, 4layer-topology.png, HDFS-6261.004.patch,
HDFS-6261.005.patch, HDFS-6261.006.patch, HDFS-6261.007.patch,
HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch
Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there
is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons)
and how to do configuration. so we need to doc it.
1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
2. Doc NodeGroup-aware properties in core-default.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[
https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557933#comment-14557933
]
Hadoop QA commented on HDFS-6261:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 1s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12660927/HDFS-6261.v3.patch |
| Optional Tests | site |
| git revision | trunk / 446d515 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/9/console |
This message was automatically generated.
Add document for enabling node group layer in HDFS
--
Key: HDFS-6261
URL: https://issues.apache.org/jira/browse/HDFS-6261
Project: Hadoop HDFS
Issue Type: Task
Components: documentation
Reporter: Wenwu Peng
Assignee: Binglin Chang
Labels: documentation
Attachments: 2-layer-topology.png, 3-layer-topology.png,
3layer-topology.png, 4layer-topology.png, HDFS-6261.004.patch,
HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch
Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there
is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons)
and how to do configuration. so we need to doc it.
1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
2. Doc NodeGroup-aware properties in core-default.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[
https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557947#comment-14557947
]
Hadoop QA commented on HDFS-6261:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735106/HDFS-6261.005.patch |
| Optional Tests | site |
| git revision | trunk / 446d515 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11121/console |
This message was automatically generated.
Add document for enabling node group layer in HDFS
--
Key: HDFS-6261
URL: https://issues.apache.org/jira/browse/HDFS-6261
Project: Hadoop HDFS
Issue Type: Task
Components: documentation
Reporter: Wenwu Peng
Assignee: Binglin Chang
Labels: documentation
Attachments: 2-layer-topology.png, 3-layer-topology.png,
3layer-topology.png, 4layer-topology.png, HDFS-6261.004.patch,
HDFS-6261.005.patch, HDFS-6261.v1.patch, HDFS-6261.v1.patch,
HDFS-6261.v2.patch, HDFS-6261.v3.patch
Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there
is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons)
and how to do configuration. so we need to doc it.
1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
2. Doc NodeGroup-aware properties in core-default.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[
https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557966#comment-14557966
]
Hadoop QA commented on HDFS-6261:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 2m 58s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 2m 56s | Site still builds. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| | | 6m 19s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735113/HDFS-6261.007.patch |
| Optional Tests | site |
| git revision | trunk / 446d515 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11123/artifact/patchprocess/whitespace.txt
|
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11123/console |
This message was automatically generated.
Add document for enabling node group layer in HDFS
--
Key: HDFS-6261
URL: https://issues.apache.org/jira/browse/HDFS-6261
Project: Hadoop HDFS
Issue Type: Task
Components: documentation
Reporter: Wenwu Peng
Assignee: Binglin Chang
Labels: documentation
Attachments: 2-layer-topology.png, 3-layer-topology.png,
3layer-topology.png, 4layer-topology.png, HDFS-6261.004.patch,
HDFS-6261.005.patch, HDFS-6261.006.patch, HDFS-6261.007.patch,
HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch
Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there
is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons)
and how to do configuration. so we need to doc it.
1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
2. Doc NodeGroup-aware properties in core-default.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[
https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557956#comment-14557956
]
Hadoop QA commented on HDFS-6261:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735109/HDFS-6261.006.patch |
| Optional Tests | site |
| git revision | trunk / 446d515 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11122/console |
This message was automatically generated.
Add document for enabling node group layer in HDFS
--
Key: HDFS-6261
URL: https://issues.apache.org/jira/browse/HDFS-6261
Project: Hadoop HDFS
Issue Type: Task
Components: documentation
Reporter: Wenwu Peng
Assignee: Binglin Chang
Labels: documentation
Attachments: 2-layer-topology.png, 3-layer-topology.png,
3layer-topology.png, 4layer-topology.png, HDFS-6261.004.patch,
HDFS-6261.005.patch, HDFS-6261.006.patch, HDFS-6261.v1.patch,
HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch
Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there
is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons)
and how to do configuration. so we need to doc it.
1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
2. Doc NodeGroup-aware properties in core-default.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[
https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557937#comment-14557937
]
Hadoop QA commented on HDFS-6261:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735103/HDFS-6261.004.patch |
| Optional Tests | site |
| git revision | trunk / 446d515 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11120/console |
This message was automatically generated.
Add document for enabling node group layer in HDFS
--
Key: HDFS-6261
URL: https://issues.apache.org/jira/browse/HDFS-6261
Project: Hadoop HDFS
Issue Type: Task
Components: documentation
Reporter: Wenwu Peng
Assignee: Binglin Chang
Labels: documentation
Attachments: 2-layer-topology.png, 3-layer-topology.png,
3layer-topology.png, 4layer-topology.png, HDFS-6261.004.patch,
HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch
Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there
is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons)
and how to do configuration. so we need to doc it.
1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
2. Doc NodeGroup-aware properties in core-default.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557957#comment-14557957 ] Allen Wittenauer commented on HDFS-6261: You can include binary files with 'git format-patch'. Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 2-layer-topology.png, 3-layer-topology.png, 3layer-topology.png, 4layer-topology.png, HDFS-6261.004.patch, HDFS-6261.005.patch, HDFS-6261.006.patch, HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14554030#comment-14554030 ] Lars Francke commented on HDFS-6261: Hey [~decster] just a friendly ping/reminder. Having this documented would be great, thank you! Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 2-layer-topology.png, 3-layer-topology.png, 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14555460#comment-14555460 ] Binglin Chang commented on HDFS-6261: - Sorry... will see if I can get this done this weekend. Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 2-layer-topology.png, 3-layer-topology.png, 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377332#comment-14377332 ] Binglin Chang commented on HDFS-6261: - Sorry for the late, will update the patch soon. Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 2-layer-topology.png, 3-layer-topology.png, 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376969#comment-14376969 ] Junping Du commented on HDFS-6261: -- Hi [~decster], given HADOOP-11495 is already get commit in, would you mind to update your patch here? I will give it a review. Thx! Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 2-layer-topology.png, 3-layer-topology.png, 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306506#comment-14306506 ] Binglin Chang commented on HDFS-6261: - bq. I'd prefer to see this get merged into the RackAwareness documentation rather than building a completely new doc OK. Will update the patch once HADOOP-11495 is resolved. Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 2-layer-topology.png, 3-layer-topology.png, 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306543#comment-14306543 ] Junping Du commented on HDFS-6261: -- Sounds like a reasonable plan, [~decster]! I may need more time for HADOOP-11495 because some other JIRA and efforts, please feel free to take that one if you can do it now and I will help to review it. Thanks [~aw] for comments also! Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 2-layer-topology.png, 3-layer-topology.png, 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290092#comment-14290092 ] Allen Wittenauer commented on HDFS-6261: I'd prefer to see this get merged into the RackAwareness documentation rather than building a completely new doc. Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 2-layer-topology.png, 3-layer-topology.png, 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[
https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290111#comment-14290111
]
Hadoop QA commented on HDFS-6261:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660927/HDFS-6261.v3.patch
against trunk revision 6c3fec5.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9319//console
This message is automatically generated.
Add document for enabling node group layer in HDFS
--
Key: HDFS-6261
URL: https://issues.apache.org/jira/browse/HDFS-6261
Project: Hadoop HDFS
Issue Type: Task
Components: documentation
Reporter: Wenwu Peng
Assignee: Binglin Chang
Labels: documentation
Attachments: 2-layer-topology.png, 3-layer-topology.png,
3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch,
HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch
Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there
is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons)
and how to do configuration. so we need to doc it.
1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
2. Doc NodeGroup-aware properties in core-default.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[
https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092565#comment-14092565
]
Hadoop QA commented on HDFS-6261:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660927/HDFS-6261.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7607//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7607//console
This message is automatically generated.
Add document for enabling node group layer in HDFS
--
Key: HDFS-6261
URL: https://issues.apache.org/jira/browse/HDFS-6261
Project: Hadoop HDFS
Issue Type: Task
Components: documentation
Reporter: Wenwu Peng
Assignee: Binglin Chang
Labels: documentation
Attachments: 2-layer-topology.png, 3-layer-topology.png,
3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch,
HDFS-6261.v1.patch, HDFS-6261.v2.patch, HDFS-6261.v3.patch
Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there
is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons)
and how to do configuration. so we need to doc it.
1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
2. Doc NodeGroup-aware properties in core-default.xml.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092448#comment-14092448 ] Wenwu Peng commented on HDFS-6261: -- Hi Junping, could you help review again? Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 2-layer-topology.png, 3-layer-topology.png, 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[
https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092452#comment-14092452
]
Hadoop QA commented on HDFS-6261:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642186/HDFS-6261.v2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7605//console
This message is automatically generated.
Add document for enabling node group layer in HDFS
--
Key: HDFS-6261
URL: https://issues.apache.org/jira/browse/HDFS-6261
Project: Hadoop HDFS
Issue Type: Task
Components: documentation
Reporter: Wenwu Peng
Assignee: Binglin Chang
Labels: documentation
Attachments: 2-layer-topology.png, 3-layer-topology.png,
3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch,
HDFS-6261.v1.patch, HDFS-6261.v2.patch
Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there
is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons)
and how to do configuration. so we need to doc it.
1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
2. Doc NodeGroup-aware properties in core-default.xml.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13985214#comment-13985214 ] Binglin Chang commented on HDFS-6261: - Sorry, looked more carefully, the failure is different, fired HDFS-6308 for this. Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 2-layer-topology.png, 3-layer-topology.png, 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13985092#comment-13985092 ] Binglin Chang commented on HDFS-6261: - The failed test is not caused by this patch, and is tracked in HDFS-5912 Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 2-layer-topology.png, 3-layer-topology.png, 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[
https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13982798#comment-13982798
]
Hadoop QA commented on HDFS-6261:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642186/HDFS-6261.v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDistributedFileSystem
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6753//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6753//console
This message is automatically generated.
Add document for enabling node group layer in HDFS
--
Key: HDFS-6261
URL: https://issues.apache.org/jira/browse/HDFS-6261
Project: Hadoop HDFS
Issue Type: Task
Components: documentation
Reporter: Wenwu Peng
Assignee: Binglin Chang
Labels: documentation
Attachments: 2-layer-topology.png, 3-layer-topology.png,
3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch,
HDFS-6261.v1.patch, HDFS-6261.v2.patch
Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there
is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons)
and how to do configuration. so we need to doc it.
1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
2. Doc NodeGroup-aware properties in core-default.xml.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13982196#comment-13982196 ] Junping Du commented on HDFS-6261: -- Thanks [~decster] for working on this effort. I quickly go through it with a few comments below (not completed): bq. The original Hadoop topology supports a 3-layer topology looks like following: I think it is better to say: previously, Hadoop only support 2-layers topology: rack and host. Mentioning datacenter layer will confuse user as it is never worked even now. For the same reason, we should mention now we support 3 layers topology/locality: rack, nodegroup and host. bq. This network topology is designed and work well for Hadoop cluster running on physical server farms. However, for Hadoop running on virtualized platform, we have additional hypervisor layer, and its characteristics include:... I think the use case of NodeGroup layer is even broader than virtualization and suitable for any sub-dependency of nodes between rack and host layer. So, it could be better to say something like This network topology is designed to work well on Hadoop cluster that only has rack (switch or power) failure dependency among nodes. However, for other cases, like: Hadoop nodes running on virtualized platform, we have additional hypervisor layer, and its characteristics include ... bq. Due to above characteristics in performance and reliability, this layer is not transparent for Hadoop... Reliability is more important here, so here better to be Due to above characteristics in reliability and performance, this layer should't be transparent for Hadoop... bq. 1st replica is on the local node or local node group of the writer For more precisely, we may say something like: 1st replica is placed on the nearest node to writer in topology. In most cases, it should be on the same node of writer, but could be on other node in the same nodegroup or rack if node of writer is not qualified (i.e. no local datanode or disk is full) to place replica. The diagram is better to omit datacenter layer according to comments above and red layer of S1 is better update to NG1 for reflecting NodeGroup layer. Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, HDFS-6261.v1.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13982746#comment-13982746 ] Binglin Chang commented on HDFS-6261: - Hi Junping, thanks for the review! I updated the patch addressing your comments. Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 2-layer-topology.png, 3-layer-topology.png, 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, HDFS-6261.v1.patch, HDFS-6261.v2.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[
https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980790#comment-13980790
]
Hadoop QA commented on HDFS-6261:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12641871/HDFS-6261.v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6728//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6728//console
This message is automatically generated.
Add document for enabling node group layer in HDFS
--
Key: HDFS-6261
URL: https://issues.apache.org/jira/browse/HDFS-6261
Project: Hadoop HDFS
Issue Type: Task
Components: documentation
Reporter: Wenwu Peng
Assignee: Binglin Chang
Labels: documentation
Attachments: 3layer-topology.png, 4layer-topology.png,
HDFS-6261.v1.patch, HDFS-6261.v1.patch
Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there
is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons)
and how to do configuration. so we need to doc it.
1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
2. Doc NodeGroup-aware properties in core-default.xml.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[
https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13979656#comment-13979656
]
Hadoop QA commented on HDFS-6261:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12641678/3layer-topology.png
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6717//console
This message is automatically generated.
Add document for enabling node group layer in HDFS
--
Key: HDFS-6261
URL: https://issues.apache.org/jira/browse/HDFS-6261
Project: Hadoop HDFS
Issue Type: Task
Components: documentation
Reporter: Wenwu Peng
Assignee: Binglin Chang
Labels: documentation
Attachments: 3layer-topology.png, 4layer-topology.png,
HDFS-6261.v1.patch
Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there
is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons)
and how to do configuration. so we need to doc it.
1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
2. Doc NodeGroup-aware properties in core-default.xml.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980720#comment-13980720 ] Binglin Chang commented on HDFS-6261: - Looks like jenkins used png image file as the patch, so it doesn't apply, resubmit patch Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Attachments: 3layer-topology.png, 4layer-topology.png, HDFS-6261.v1.patch, HDFS-6261.v1.patch Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6261) Add document for enabling node group layer in HDFS
[ https://issues.apache.org/jira/browse/HDFS-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13974349#comment-13974349 ] Junping Du commented on HDFS-6261: -- NodeGroup Topology feature is completed in HDFS part, but haven't finished in YARN (pending on YARN-18 and YARN-19). Agree we can do HDFS first, so move this JIRA to HDFS. Add document for enabling node group layer in HDFS -- Key: HDFS-6261 URL: https://issues.apache.org/jira/browse/HDFS-6261 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Wenwu Peng Assignee: Binglin Chang Labels: documentation Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it. 1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current 2. Doc NodeGroup-aware properties in core-default.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
