[jira] [Updated] (HDFS-13448) HDFS Block Placement - Ignore Locality for First Block Replica
[
https://issues.apache.org/jira/browse/HDFS-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-13448:
---
Status: Patch Available (was: Open)
Updated the patch to add another configurable valve that allows a cluster admin
to disable this feature. However, I think this should be a feature allowed by
default. This HDFS behavior therefore should not an extension of
{{BlockPlacementPolicyDefault}}, but a part of it.
> HDFS Block Placement - Ignore Locality for First Block Replica
> --
>
> Key: HDFS-13448
> URL: https://issues.apache.org/jira/browse/HDFS-13448
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: block placement, hdfs-client
>Affects Versions: 3.0.1, 2.9.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-13448.1.patch, HDFS-13448.2.patch
>
>
> According to the HDFS Block Place Rules:
> {quote}
> /**
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on a different node of the rack as the second replica.
> */
> {quote}
> However, there is a hint for the hdfs-client that allows the block placement
> request to not put a block replica on the local datanode _where 'local' means
> the same host as the client is being run on._
> {quote}
> /**
>* Advise that a block replica NOT be written to the local DataNode where
>* 'local' means the same host as the client is being run on.
>*
>* @see CreateFlag#NO_LOCAL_WRITE
>*/
> {quote}
> I propose that we add a new flag that allows the hdfs-client to request that
> the first block replica be placed on a random DataNode in the cluster. The
> subsequent block replicas should follow the normal block placement rules.
> The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block
> replica is not placed on the local node, but it is still placed on the local
> rack. Where this comes into play is where you have, for example, a flume
> agent that is loading data into HDFS.
> If the Flume agent is running on a DataNode, then by default, the DataNode
> local to the Flume agent will always get the first block replica and this
> leads to un-even block placements, with the local node always filling up
> faster than any other node in the cluster.
> Modifying this example, if the DataNode is removed from the host where the
> Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then
> the default block placement policy will still prefer the local rack. This
> remedies the situation only so far as now the first block replica will always
> be distributed to a DataNode on the local rack.
> This new flag would allow a single Flume agent to distribute the blocks
> randomly, evenly, over the entire cluster instead of hot-spotting the local
> node or the local rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13448) HDFS Block Placement - Ignore Locality for First Block Replica
[
https://issues.apache.org/jira/browse/HDFS-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-13448:
---
Attachment: HDFS-13448.2.patch
> HDFS Block Placement - Ignore Locality for First Block Replica
> --
>
> Key: HDFS-13448
> URL: https://issues.apache.org/jira/browse/HDFS-13448
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: block placement, hdfs-client
>Affects Versions: 2.9.0, 3.0.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-13448.1.patch, HDFS-13448.2.patch
>
>
> According to the HDFS Block Place Rules:
> {quote}
> /**
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on a different node of the rack as the second replica.
> */
> {quote}
> However, there is a hint for the hdfs-client that allows the block placement
> request to not put a block replica on the local datanode _where 'local' means
> the same host as the client is being run on._
> {quote}
> /**
>* Advise that a block replica NOT be written to the local DataNode where
>* 'local' means the same host as the client is being run on.
>*
>* @see CreateFlag#NO_LOCAL_WRITE
>*/
> {quote}
> I propose that we add a new flag that allows the hdfs-client to request that
> the first block replica be placed on a random DataNode in the cluster. The
> subsequent block replicas should follow the normal block placement rules.
> The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block
> replica is not placed on the local node, but it is still placed on the local
> rack. Where this comes into play is where you have, for example, a flume
> agent that is loading data into HDFS.
> If the Flume agent is running on a DataNode, then by default, the DataNode
> local to the Flume agent will always get the first block replica and this
> leads to un-even block placements, with the local node always filling up
> faster than any other node in the cluster.
> Modifying this example, if the DataNode is removed from the host where the
> Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then
> the default block placement policy will still prefer the local rack. This
> remedies the situation only so far as now the first block replica will always
> be distributed to a DataNode on the local rack.
> This new flag would allow a single Flume agent to distribute the blocks
> randomly, evenly, over the entire cluster instead of hot-spotting the local
> node or the local rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13448) HDFS Block Placement - Ignore Locality for First Block Replica
[
https://issues.apache.org/jira/browse/HDFS-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-13448:
---
Status: Open (was: Patch Available)
> HDFS Block Placement - Ignore Locality for First Block Replica
> --
>
> Key: HDFS-13448
> URL: https://issues.apache.org/jira/browse/HDFS-13448
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: block placement, hdfs-client
>Affects Versions: 3.0.1, 2.9.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-13448.1.patch, HDFS-13448.2.patch
>
>
> According to the HDFS Block Place Rules:
> {quote}
> /**
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on a different node of the rack as the second replica.
> */
> {quote}
> However, there is a hint for the hdfs-client that allows the block placement
> request to not put a block replica on the local datanode _where 'local' means
> the same host as the client is being run on._
> {quote}
> /**
>* Advise that a block replica NOT be written to the local DataNode where
>* 'local' means the same host as the client is being run on.
>*
>* @see CreateFlag#NO_LOCAL_WRITE
>*/
> {quote}
> I propose that we add a new flag that allows the hdfs-client to request that
> the first block replica be placed on a random DataNode in the cluster. The
> subsequent block replicas should follow the normal block placement rules.
> The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block
> replica is not placed on the local node, but it is still placed on the local
> rack. Where this comes into play is where you have, for example, a flume
> agent that is loading data into HDFS.
> If the Flume agent is running on a DataNode, then by default, the DataNode
> local to the Flume agent will always get the first block replica and this
> leads to un-even block placements, with the local node always filling up
> faster than any other node in the cluster.
> Modifying this example, if the DataNode is removed from the host where the
> Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then
> the default block placement policy will still prefer the local rack. This
> remedies the situation only so far as now the first block replica will always
> be distributed to a DataNode on the local rack.
> This new flag would allow a single Flume agent to distribute the blocks
> randomly, evenly, over the entire cluster instead of hot-spotting the local
> node or the local rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-13326:
---
Description:
>From DFSRouterAdmin cmd, currently the update logic is implemented inside add
>operation, where it has some limitation (e.g. cannot update "readonly" or
>removing a destination). Given the RPC alreadys separate add and update
>operations, it would be better to do the same in cmd level.
was:
1. From DFSRouterAdmin cmd, currently the update logic is implemented inside
add operation, where it has some limitation (e.g. cannot update "readonly" or
removing a destination). Given the RPC alreadys separate add and update
operations, it would be better to do the same in cmd level.
2. Currently in the MountTable tab, the "readonly" field always show empty, no
matter whether the mount entry is readonly or not. From the code perspective,
it tries to show:
{code:java}
{code}
The federationhealth.html will load hadoop.css, however the hadoop.css doesn't
have classes with a prefix "dfshealth-mount-read-only". This could be fixed in
HDFS-13204.
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Gang Li
>Priority: Minor
> Attachments: HDFS-13326.000.patch, HDFS-13326.001.patch
>
>
> From DFSRouterAdmin cmd, currently the update logic is implemented inside add
> operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438139#comment-16438139
]
genericqa commented on HDFS-13326:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
4s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 49s{color} | {color:orange} hadoop-hdfs-project: The patch generated 12 new
+ 0 unchanged - 0 fixed = 12 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 59s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 53s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
40s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestReconstructStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13326 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919012/HDFS-13326.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b4b94011abc7 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 995cba6 |
|
[jira] [Updated] (HDFS-13330) ShortCircuitCache#fetchOrCreate never retries
[
https://issues.apache.org/jira/browse/HDFS-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-13330:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> ShortCircuitCache#fetchOrCreate never retries
> -
>
> Key: HDFS-13330
> URL: https://issues.apache.org/jira/browse/HDFS-13330
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0, 3.1.1, 3.0.3
>
> Attachments: HDFS-13330.001.patch, HDFS-13330.002.patch,
> HDFS-13330.004.patch, HDFS-13330.005.patch, HDFS-13330.branch-2.patch
>
>
> The follow do .. while(false) loop seems useless to me. The code intended to
> retry but it never worked. Let's fix it.
> {code:java:title=ShortCircuitCache#fetchOrCreate}
> ShortCircuitReplicaInfo info = null;
> do {
> if (closed) {
> LOG.trace("{}: can't fethchOrCreate {} because the cache is closed.",
> this, key);
> return null;
> }
> Waitable waitable = replicaInfoMap.get(key);
> if (waitable != null) {
> try {
> info = fetch(key, waitable);
> } catch (RetriableException e) {
> LOG.debug("{}: retrying {}", this, e.getMessage());
> }
> }
> } while (false);{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13448) HDFS Block Placement - Ignore Locality for First Block Replica
[
https://issues.apache.org/jira/browse/HDFS-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438028#comment-16438028
]
genericqa commented on HDFS-13448:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 31m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 28m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
25s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
36s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 79m
26s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}236m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13448 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918991/HDFS-13448.1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 06870f4f2ca1 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e66e287 |
|
[jira] [Comment Edited] (HDFS-13441) DataNode missed BlockKey update from NameNode due to HeartbeatResponse was dropped
[
https://issues.apache.org/jira/browse/HDFS-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437740#comment-16437740
]
yunjiong zhao edited comment on HDFS-13441 at 4/13/18 9:50 PM:
---
{quote}
BlockKey is usually synchronized aggressively – IIRC every 2.5 hours, and
BlockKey's life time is much longer than that (can't recall right away) so it's
surprising to me a single missing heartbeat would cause the error you
mentioned. There's probably something deeper you need to dig into.
{quote}
DataNode gets block keys from NameNode happens at two places.
One is when DataNode registers to NameNode.
Another one is via heartbeat, by default happens every 600 minutes.
By default,
{quote}
dfs.block.access.key.update.interval
600
Interval in minutes at which namenode updates its access keys.
{quote}
{quote}
dfs.block.access.token.lifetime
600
The lifetime of access tokens in minutes.
{quote}
[~jojochuang] , double checked the code and log, the DataNode must be missing
*two* heartbeats from Standby NameNode which contains new block key just before
Standby NameNode become active.
We have more than 2000 Datanodes in this cluster, if more than three Datanodes
doesn't have the current Block key, worst case is users will not able to read
some blocks for 10 at most hours.
was (Author: zhaoyunjiong):
{quote}
BlockKey is usually synchronized aggressively – IIRC every 2.5 hours, and
BlockKey's life time is much longer than that (can't recall right away) so it's
surprising to me a single missing heartbeat would cause the error you
mentioned. There's probably something deeper you need to dig into.
{quote}
DataNode gets block keys from NameNode happens at two places.
One is when DataNode registers to NameNode.
Another one is via heartbeat, by default happens every 600 minutes.
By default,
{quote}
dfs.block.access.key.update.interval
600
Interval in minutes at which namenode updates its access keys.
{quote}
{quote}
dfs.block.access.token.lifetime
600
The lifetime of access tokens in minutes.
{quote}
Not any single heartbeat of course.
The missing heartbeat must from Standby NameNode which contains new block key,
and Standby NameNode transferred to Active NameNode after that missing
heartbeat happened and before the new heartbeat which contains new block keys.
We have more than 2000 Datanodes in this cluster, if more than three Datanodes
doesn't have the current Block key, worst case is users will not able to read
some blocks for 10 hours.
> DataNode missed BlockKey update from NameNode due to HeartbeatResponse was
> dropped
> --
>
> Key: HDFS-13441
> URL: https://issues.apache.org/jira/browse/HDFS-13441
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 2.7.1
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Attachments: HDFS-13441.patch
>
>
> After NameNode failover, lots of application failed due to some DataNodes
> can't re-compute password from block token.
> {code:java}
> 2018-04-11 20:10:52,448 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> hdc3-lvs01-400-1701-048.stratus.lvs.ebay.com:50010:DataXceiver error
> processing unknown operation src: /10.142.74.116:57404 dst:
> /10.142.77.45:50010
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
> [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=1523538652448,
> keyId=1762737944, userId=hadoop,
> blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
> access modes=[WRITE]), since the required block key (keyID=1762737944)
> doesn't exist.]
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:376)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getSaslStreams(SaslDataTransferServer.java:300)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:127)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:194)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password
[jira] [Updated] (HDFS-13441) DataNode missed BlockKey update from NameNode due to HeartbeatResponse was dropped
[
https://issues.apache.org/jira/browse/HDFS-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yunjiong zhao updated HDFS-13441:
-
Description:
After NameNode failover, lots of application failed due to some DataNodes can't
re-compute password from block token.
{code:java}
2018-04-11 20:10:52,448 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
hdc3-lvs01-400-1701-048.stratus.lvs.ebay.com:50010:DataXceiver error processing
unknown operation src: /10.142.74.116:57404 dst: /10.142.77.45:50010
javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
[Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
re-compute password for block_token_identifier (expiryDate=1523538652448,
keyId=1762737944, userId=hadoop,
blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
access modes=[WRITE]), since the required block key (keyID=1762737944) doesn't
exist.]
at
com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
at
com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:376)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getSaslStreams(SaslDataTransferServer.java:300)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:127)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:194)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
re-compute password for block_token_identifier (expiryDate=1523538652448,
keyId=1762737944, userId=hadoop,
blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
access modes=[WRITE]), since the required block key (keyID=1762737944) doesn't
exist.
at
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrievePassword(BlockTokenSecretManager.java:382)
at
org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrievePassword(BlockPoolTokenSecretManager.java:79)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.buildServerPassword(SaslDataTransferServer.java:318)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$100(SaslDataTransferServer.java:73)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$2.apply(SaslDataTransferServer.java:297)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:241)
at
com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
... 7 more
{code}
In the DataNode log, we didn't see DataNode update block keys around 2018-04-11
09:55:00 and around 2018-04-11 19:55:00.
{code:java}
2018-04-10 14:51:36,424 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
2018-04-10 23:55:38,420 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
2018-04-11 00:51:34,792 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
2018-04-11 10:51:39,403 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
2018-04-11 20:51:44,422 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
2018-04-12 02:54:47,855 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
2018-04-12 05:55:44,456 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
{code}
The reason is there is SocketTimeOutException when sending heartbeat to
StandbyNameNode
{code:java}
2018-04-11 09:55:34,699 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
IOException in offerService
java.net.SocketTimeoutException: Call From
hdc3-lvs01-400-1701-048.stratus.lvs.ebay.com/10.142.77.45 to
ares-nn.vip.ebay.com:8030 failed on socket timeout exception:
java.net.SocketTimeoutException: 6 millis timeout while waiting for channel
to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.142.77.45:48803 remote=ares-nn.vip.ebay.com/10.103.108.200:8030]; For
more details see: http://wiki.apache.org/hadoop/SocketTimeout
at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown
Source)
at
[jira] [Commented] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437968#comment-16437968
]
Gang Li commented on HDFS-13326:
I just uploaded [^HDFS-13326.001.patch] supporting new add and update in the
command line based on [~ywskycn]'s comments.
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Gang Li
>Priority: Minor
> Attachments: HDFS-13326.000.patch, HDFS-13326.001.patch
>
>
> 1. From DFSRouterAdmin cmd, currently the update logic is implemented inside
> add operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
> 2. Currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only". This could be
> fixed in HDFS-13204.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13326) RBF: Improve the interfaces to modify and view mount tables
[
https://issues.apache.org/jira/browse/HDFS-13326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gang Li updated HDFS-13326:
---
Attachment: HDFS-13326.001.patch
> RBF: Improve the interfaces to modify and view mount tables
> ---
>
> Key: HDFS-13326
> URL: https://issues.apache.org/jira/browse/HDFS-13326
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Gang Li
>Priority: Minor
> Attachments: HDFS-13326.000.patch, HDFS-13326.001.patch
>
>
> 1. From DFSRouterAdmin cmd, currently the update logic is implemented inside
> add operation, where it has some limitation (e.g. cannot update "readonly" or
> removing a destination). Given the RPC alreadys separate add and update
> operations, it would be better to do the same in cmd level.
> 2. Currently in the MountTable tab, the "readonly" field always show empty,
> no matter whether the mount entry is readonly or not. From the code
> perspective, it tries to show:
> {code:java}
> {code}
> The federationhealth.html will load hadoop.css, however the hadoop.css
> doesn't have classes with a prefix "dfshealth-mount-read-only". This could be
> fixed in HDFS-13204.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13441) DataNode missed BlockKey update from NameNode due to HeartbeatResponse was dropped
[
https://issues.apache.org/jira/browse/HDFS-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yunjiong zhao updated HDFS-13441:
-
Description:
After NameNode failover, lots of application failed due to some DataNodes can't
re-compute password from block token.
{code:java}
2018-04-11 20:10:52,448 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
hdc3-lvs01-400-1701-048.stratus.lvs.ebay.com:50010:DataXceiver error processing
unknown operation src: /10.142.74.116:57404 dst: /10.142.77.45:50010
javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
[Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
re-compute password for block_token_identifier (expiryDate=1523538652448,
keyId=1762737944, userId=hadoop,
blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
access modes=[WRITE]), since the required block key (keyID=1762737944) doesn't
exist.]
at
com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
at
com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:376)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getSaslStreams(SaslDataTransferServer.java:300)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:127)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:194)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
re-compute password for block_token_identifier (expiryDate=1523538652448,
keyId=1762737944, userId=hadoop,
blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
access modes=[WRITE]), since the required block key (keyID=1762737944) doesn't
exist.
at
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrievePassword(BlockTokenSecretManager.java:382)
at
org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrievePassword(BlockPoolTokenSecretManager.java:79)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.buildServerPassword(SaslDataTransferServer.java:318)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$100(SaslDataTransferServer.java:73)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$2.apply(SaslDataTransferServer.java:297)
at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:241)
at
com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589)
... 7 more
{code}
In the DataNode log, we didn't see DataNode update block keys around 2018-04-11
09:55:00 and around 2018-04-11 19:55:00.
{code:java}
2018-04-10 14:51:36,424 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
2018-04-10 23:55:38,420 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
2018-04-11 00:51:34,792 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
2018-04-11 10:51:39,403 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
2018-04-11 20:51:44,422 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
2018-04-12 02:54:47,855 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
2018-04-12 05:55:44,456 INFO
org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager: Setting
block keys
{code}
The reason is there is SocketTimeOutException when sending heartbeat to
StandbyNameNode
{code:java}
2018-04-11 09:55:34,699 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
IOException in offerService
java.net.SocketTimeoutException: Call From
hdc3-lvs01-400-1701-048.stratus.lvs.ebay.com/10.142.77.45 to
ares-nn.vip.ebay.com:8030 failed on socket timeout exception:
java.net.SocketTimeoutException: 6 millis timeout while waiting for channel
to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/10.142.77.45:48803 remote=ares-nn.vip.ebay.com/10.103.108.200:8030]; For
more details see: http://wiki.apache.org/hadoop/SocketTimeout
at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown
Source)
at
[jira] [Commented] (HDFS-13445) Hostname is not getting resolved when we pass IP in webhdfs request.
[ https://issues.apache.org/jira/browse/HDFS-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437907#comment-16437907 ] Ajay Kumar commented on HDFS-13445: --- [~RANith], thanks for working on patch, mind creating a test case? > Hostname is not getting resolved when we pass IP in webhdfs request. > > > Key: HDFS-13445 > URL: https://issues.apache.org/jira/browse/HDFS-13445 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, security, webhdfs >Affects Versions: 2.8.3 >Reporter: Ranith Sardar >Assignee: Ranith Sardar >Priority: Major > Attachments: HDFS-13445.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13448) HDFS Block Placement - Ignore Locality for First Block Replica
[
https://issues.apache.org/jira/browse/HDFS-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437904#comment-16437904
]
Ajay Kumar commented on HDFS-13448:
---
Hi [~belugabehr], this looks like a valid use case. Had a quick look at patch,
i suggest instead of changing defaultblockplacement policy create a new policy
(by extending it) which works with new flag. Also enable, disable that flag
according to some configurable property.
> HDFS Block Placement - Ignore Locality for First Block Replica
> --
>
> Key: HDFS-13448
> URL: https://issues.apache.org/jira/browse/HDFS-13448
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: block placement, hdfs-client
>Affects Versions: 2.9.0, 3.0.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-13448.1.patch
>
>
> According to the HDFS Block Place Rules:
> {quote}
> /**
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on a different node of the rack as the second replica.
> */
> {quote}
> However, there is a hint for the hdfs-client that allows the block placement
> request to not put a block replica on the local datanode _where 'local' means
> the same host as the client is being run on._
> {quote}
> /**
>* Advise that a block replica NOT be written to the local DataNode where
>* 'local' means the same host as the client is being run on.
>*
>* @see CreateFlag#NO_LOCAL_WRITE
>*/
> {quote}
> I propose that we add a new flag that allows the hdfs-client to request that
> the first block replica be placed on a random DataNode in the cluster. The
> subsequent block replicas should follow the normal block placement rules.
> The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block
> replica is not placed on the local node, but it is still placed on the local
> rack. Where this comes into play is where you have, for example, a flume
> agent that is loading data into HDFS.
> If the Flume agent is running on a DataNode, then by default, the DataNode
> local to the Flume agent will always get the first block replica and this
> leads to un-even block placements, with the local node always filling up
> faster than any other node in the cluster.
> Modifying this example, if the DataNode is removed from the host where the
> Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then
> the default block placement policy will still prefer the local rack. This
> remedies the situation only so far as now the first block replica will always
> be distributed to a DataNode on the local rack.
> This new flag would allow a single Flume agent to distribute the blocks
> randomly, evenly, over the entire cluster instead of hot-spotting the local
> node or the local rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13197) Ozone: Fix ConfServlet#getOzoneTags cmd
[ https://issues.apache.org/jira/browse/HDFS-13197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13197: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks [~ajayydv] for the contribution. I've committed the fix into the feature branch. > Ozone: Fix ConfServlet#getOzoneTags cmd > --- > > Key: HDFS-13197 > URL: https://issues.apache.org/jira/browse/HDFS-13197 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-13197-HDFS-7240.000.patch, > HDFS-13197-HDFS-7240.001.patch, HDFS-13197-HDFS-7240.002.patch, > HDFS-13197-HDFS-7240.003.patch, Screen Shot 2018-04-10 at 2.05.35 PM.png > > > This is broken after merging trunk change HADOOP-15007 into HDFS-7240 branch. > I remove the cmd and related test to have a clean merge. [~ajakumar], please > fix the cmd and bring back the related test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13197) Ozone: Fix ConfServlet#getOzoneTags cmd
[ https://issues.apache.org/jira/browse/HDFS-13197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13197: -- Summary: Ozone: Fix ConfServlet#getOzoneTags cmd (was: Ozone: Fix ConfServlet#getOzoneTags cmd after HADOOP-15007) > Ozone: Fix ConfServlet#getOzoneTags cmd > --- > > Key: HDFS-13197 > URL: https://issues.apache.org/jira/browse/HDFS-13197 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-13197-HDFS-7240.000.patch, > HDFS-13197-HDFS-7240.001.patch, HDFS-13197-HDFS-7240.002.patch, > HDFS-13197-HDFS-7240.003.patch, Screen Shot 2018-04-10 at 2.05.35 PM.png > > > This is broken after merging trunk change HADOOP-15007 into HDFS-7240 branch. > I remove the cmd and related test to have a clean merge. [~ajakumar], please > fix the cmd and bring back the related test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13447) Fix Typos - Node Not Chosen
[
https://issues.apache.org/jira/browse/HDFS-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437870#comment-16437870
]
genericqa commented on HDFS-13447:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 56s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13447 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918978/HDFS-13447.1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 648700afef15 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e66e287 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23928/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-13422) Ozone: Fix whitespaces and license issues in HDFS-7240 branch
[ https://issues.apache.org/jira/browse/HDFS-13422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437865#comment-16437865 ] Xiaoyu Yao commented on HDFS-13422: --- Thanks [~ljain] for working on this. With the latest patch, I still see the asf license issues in the Jenkins report: https://builds.apache.org/job/PreCommit-HDFS-Build/23924/artifact/out/patch-asflicense-problems.txt. Can you fix it with the next patch? > Ozone: Fix whitespaces and license issues in HDFS-7240 branch > - > > Key: HDFS-13422 > URL: https://issues.apache.org/jira/browse/HDFS-13422 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-13422-HDFS-7240.001.patch > > > This jira will be used to fix various findbugs, javac, license and findbugs > issues in HDFS-7240 branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12828) OIV ReverseXML Processor Fails With Escaped Characters
[
https://issues.apache.org/jira/browse/HDFS-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437851#comment-16437851
]
Erik Krogen commented on HDFS-12828:
Unit test failures look unrelated...
> OIV ReverseXML Processor Fails With Escaped Characters
> --
>
> Key: HDFS-12828
> URL: https://issues.apache.org/jira/browse/HDFS-12828
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-12828.000.patch, fsimage_008.xml
>
>
> The HDFS OIV ReverseXML processor fails if the XML file contains escaped
> characters:
> {code}
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs dfs -fs hdfs://localhost:9000/ -ls /
> Found 4 items
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:48 /foo
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:49 /foo"
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:50 /foo`
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:49 /foo&
> {code}
> Then after doing {{saveNamespace}} on that NameNode...
> {code}
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs oiv -i
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008 -o
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml -p XML
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs oiv -i
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml -o
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml.rev -p
> ReverseXML
> OfflineImageReconstructor failed: unterminated entity ref starting with &
> org.apache.hadoop.hdfs.util.XMLUtils$UnmanglingError: unterminated entity ref
> starting with &
> at
> org.apache.hadoop.hdfs.util.XMLUtils.unmangleXmlString(XMLUtils.java:232)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildrenHelper(OfflineImageReconstructor.java:383)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildrenHelper(OfflineImageReconstructor.java:379)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildren(OfflineImageReconstructor.java:418)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.access$1000(OfflineImageReconstructor.java:95)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$INodeSectionProcessor.process(OfflineImageReconstructor.java:524)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1710)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1765)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:191)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:134)
> {code}
> See attachments for relevant fsimage XML file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12828) OIV ReverseXML Processor Fails With Escaped Characters
[
https://issues.apache.org/jira/browse/HDFS-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437845#comment-16437845
]
genericqa commented on HDFS-12828:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 6s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-12828 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918981/HDFS-12828.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 775eea79b119 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e66e287 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23929/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23929/testReport/ |
| Max. process+thread count | 3492 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-13197) Ozone: Fix ConfServlet#getOzoneTags cmd after HADOOP-15007
[
https://issues.apache.org/jira/browse/HDFS-13197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437828#comment-16437828
]
genericqa commented on HDFS-13197:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
15s{color} | {color:red} framework in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
57s{color} | {color:red} hadoop-hdds/common in HDFS-7240 has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} framework in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
15s{color} | {color:red} framework in HDFS-7240 failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
12s{color} | {color:red} framework in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 16s{color} | {color:orange} hadoop-hdds: The patch generated 21 new + 0
unchanged - 0 fixed = 21 total (was 0) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
11s{color} | {color:red} framework in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} framework in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
13s{color} | {color:red} framework in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
57s{color} | {color:green} common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 12s{color}
| {color:red} framework in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
24s{color} | {color:red} The patch generated 66 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:b78c94f |
| JIRA Issue | HDFS-13197 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918985/HDFS-13197-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit
[jira] [Commented] (HDFS-13439) Add test case for read block operation when it is moved
[ https://issues.apache.org/jira/browse/HDFS-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437824#comment-16437824 ] Arpit Agarwal commented on HDFS-13439: -- Thanks for the test case [~ajayydv]. Can you also add a check at the end to verify that after the client closes the file, the source block is not on the disk anymore and the destination block is there? And also a second reader can successfully read the block from its new location? > Add test case for read block operation when it is moved > --- > > Key: HDFS-13439 > URL: https://issues.apache.org/jira/browse/HDFS-13439 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13439.00.patch, HDFS-13439.01.patch > > > Add test case for read block operation when it is moved -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13330) ShortCircuitCache#fetchOrCreate never retries
[
https://issues.apache.org/jira/browse/HDFS-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-13330:
---
Fix Version/s: 3.0.3
> ShortCircuitCache#fetchOrCreate never retries
> -
>
> Key: HDFS-13330
> URL: https://issues.apache.org/jira/browse/HDFS-13330
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0, 3.1.1, 3.0.3
>
> Attachments: HDFS-13330.001.patch, HDFS-13330.002.patch,
> HDFS-13330.004.patch, HDFS-13330.005.patch, HDFS-13330.branch-2.patch
>
>
> The follow do .. while(false) loop seems useless to me. The code intended to
> retry but it never worked. Let's fix it.
> {code:java:title=ShortCircuitCache#fetchOrCreate}
> ShortCircuitReplicaInfo info = null;
> do {
> if (closed) {
> LOG.trace("{}: can't fethchOrCreate {} because the cache is closed.",
> this, key);
> return null;
> }
> Waitable waitable = replicaInfoMap.get(key);
> if (waitable != null) {
> try {
> info = fetch(key, waitable);
> } catch (RetriableException e) {
> LOG.debug("{}: retrying {}", this, e.getMessage());
> }
> }
> } while (false);{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13330) ShortCircuitCache#fetchOrCreate never retries
[
https://issues.apache.org/jira/browse/HDFS-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-13330:
---
Fix Version/s: 3.1.1
> ShortCircuitCache#fetchOrCreate never retries
> -
>
> Key: HDFS-13330
> URL: https://issues.apache.org/jira/browse/HDFS-13330
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: HDFS-13330.001.patch, HDFS-13330.002.patch,
> HDFS-13330.004.patch, HDFS-13330.005.patch, HDFS-13330.branch-2.patch
>
>
> The follow do .. while(false) loop seems useless to me. The code intended to
> retry but it never worked. Let's fix it.
> {code:java:title=ShortCircuitCache#fetchOrCreate}
> ShortCircuitReplicaInfo info = null;
> do {
> if (closed) {
> LOG.trace("{}: can't fethchOrCreate {} because the cache is closed.",
> this, key);
> return null;
> }
> Waitable waitable = replicaInfoMap.get(key);
> if (waitable != null) {
> try {
> info = fetch(key, waitable);
> } catch (RetriableException e) {
> LOG.debug("{}: retrying {}", this, e.getMessage());
> }
> }
> } while (false);{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13433) webhdfs requests can be routed incorrectly in federated cluster
[
https://issues.apache.org/jira/browse/HDFS-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437772#comment-16437772
]
genericqa commented on HDFS-13433:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 114
unchanged - 1 fixed = 114 total (was 115) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
37s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}112m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}218m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestSafeMode |
| | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13433 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918963/HDFS-13433.04.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient
[jira] [Commented] (HDFS-13441) DataNode missed BlockKey update from NameNode due to HeartbeatResponse was dropped
[
https://issues.apache.org/jira/browse/HDFS-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437740#comment-16437740
]
yunjiong zhao commented on HDFS-13441:
--
{quote}
BlockKey is usually synchronized aggressively – IIRC every 2.5 hours, and
BlockKey's life time is much longer than that (can't recall right away) so it's
surprising to me a single missing heartbeat would cause the error you
mentioned. There's probably something deeper you need to dig into.
{quote}
DataNode gets block keys from NameNode happens at two places.
One is when DataNode registers to NameNode.
Another one is via heartbeat, by default happens every 600 minutes.
By default,
{quote}
dfs.block.access.key.update.interval
600
Interval in minutes at which namenode updates its access keys.
{quote}
{quote}
dfs.block.access.token.lifetime
600
The lifetime of access tokens in minutes.
{quote}
Not any single heartbeat of course.
The missing heartbeat must from Standby NameNode which contains new block key,
and Standby NameNode transferred to Active NameNode after that missing
heartbeat happened and before the new heartbeat which contains new block keys.
We have more than 2000 Datanodes in this cluster, if more than three Datanodes
doesn't have the current Block key, worst case is users will not able to read
some blocks for 10 hours.
> DataNode missed BlockKey update from NameNode due to HeartbeatResponse was
> dropped
> --
>
> Key: HDFS-13441
> URL: https://issues.apache.org/jira/browse/HDFS-13441
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 2.7.1
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Attachments: HDFS-13441.patch
>
>
> After NameNode failover, lots of application failed due to some DataNodes
> can't re-compute password from block token.
> {code:java}
> 2018-04-11 20:10:52,448 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> hdc3-lvs01-400-1701-048.stratus.lvs.ebay.com:50010:DataXceiver error
> processing unknown operation src: /10.142.74.116:57404 dst:
> /10.142.77.45:50010
> javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password
> [Caused by org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=1523538652448,
> keyId=1762737944, userId=hadoop,
> blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
> access modes=[WRITE]), since the required block key (keyID=1762737944)
> doesn't exist.]
> at
> com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598)
> at
> com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslParticipant.evaluateChallengeOrResponse(SaslParticipant.java:115)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:376)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getSaslStreams(SaslDataTransferServer.java:300)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:127)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:194)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: Can't
> re-compute password for block_token_identifier (expiryDate=1523538652448,
> keyId=1762737944, userId=hadoop,
> blockPoolId=BP-36315570-10.103.108.13-1423055488042, blockId=12142862700,
> access modes=[WRITE]), since the required block key (keyID=1762737944)
> doesn't exist.
> at
> org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.retrievePassword(BlockTokenSecretManager.java:382)
> at
> org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.retrievePassword(BlockPoolTokenSecretManager.java:79)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.buildServerPassword(SaslDataTransferServer.java:318)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.access$100(SaslDataTransferServer.java:73)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$2.apply(SaslDataTransferServer.java:297)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer$SaslServerCallbackHandler.handle(SaslDataTransferServer.java:241)
> at
>
[jira] [Updated] (HDFS-13448) HDFS Block Placement - Ignore Locality
[
https://issues.apache.org/jira/browse/HDFS-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-13448:
---
Status: Patch Available (was: Open)
> HDFS Block Placement - Ignore Locality
> --
>
> Key: HDFS-13448
> URL: https://issues.apache.org/jira/browse/HDFS-13448
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: block placement, hdfs-client
>Affects Versions: 3.0.1, 2.9.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-13448.1.patch
>
>
> According to the HDFS Block Place Rules:
> {quote}
> /**
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on a different node of the rack as the second replica.
> */
> {quote}
> However, there is a hint for the hdfs-client that allows the block placement
> request to not put a block replica on the local datanode _where 'local' means
> the same host as the client is being run on._
> {quote}
> /**
>* Advise that a block replica NOT be written to the local DataNode where
>* 'local' means the same host as the client is being run on.
>*
>* @see CreateFlag#NO_LOCAL_WRITE
>*/
> {quote}
> I propose that we add a new flag that allows the hdfs-client to request that
> the first block replica be placed on a random DataNode in the cluster. The
> subsequent block replicas should follow the normal block placement rules.
> The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block
> replica is not placed on the local node, but it is still placed on the local
> rack. Where this comes into play is where you have, for example, a flume
> agent that is loading data into HDFS.
> If the Flume agent is running on a DataNode, then by default, the DataNode
> local to the Flume agent will always get the first block replica and this
> leads to un-even block placements, with the local node always filling up
> faster than any other node in the cluster.
> Modifying this example, if the DataNode is removed from the host where the
> Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then
> the default block placement policy will still prefer the local rack. This
> remedies the situation only so far as now the first block replica will always
> be distributed to a DataNode on the local rack.
> This new flag would allow a single Flume agent to distribute the blocks
> randomly, evenly, over the entire cluster instead of hot-spotting the local
> node or the local rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13448) HDFS Block Placement - Ignore Locality for First Block Replica
[
https://issues.apache.org/jira/browse/HDFS-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-13448:
---
Summary: HDFS Block Placement - Ignore Locality for First Block Replica
(was: HDFS Block Placement - Ignore Locality)
> HDFS Block Placement - Ignore Locality for First Block Replica
> --
>
> Key: HDFS-13448
> URL: https://issues.apache.org/jira/browse/HDFS-13448
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: block placement, hdfs-client
>Affects Versions: 2.9.0, 3.0.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-13448.1.patch
>
>
> According to the HDFS Block Place Rules:
> {quote}
> /**
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on a different node of the rack as the second replica.
> */
> {quote}
> However, there is a hint for the hdfs-client that allows the block placement
> request to not put a block replica on the local datanode _where 'local' means
> the same host as the client is being run on._
> {quote}
> /**
>* Advise that a block replica NOT be written to the local DataNode where
>* 'local' means the same host as the client is being run on.
>*
>* @see CreateFlag#NO_LOCAL_WRITE
>*/
> {quote}
> I propose that we add a new flag that allows the hdfs-client to request that
> the first block replica be placed on a random DataNode in the cluster. The
> subsequent block replicas should follow the normal block placement rules.
> The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block
> replica is not placed on the local node, but it is still placed on the local
> rack. Where this comes into play is where you have, for example, a flume
> agent that is loading data into HDFS.
> If the Flume agent is running on a DataNode, then by default, the DataNode
> local to the Flume agent will always get the first block replica and this
> leads to un-even block placements, with the local node always filling up
> faster than any other node in the cluster.
> Modifying this example, if the DataNode is removed from the host where the
> Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then
> the default block placement policy will still prefer the local rack. This
> remedies the situation only so far as now the first block replica will always
> be distributed to a DataNode on the local rack.
> This new flag would allow a single Flume agent to distribute the blocks
> randomly, evenly, over the entire cluster instead of hot-spotting the local
> node or the local rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13448) HDFS Block Placement - Ignore Locality
[
https://issues.apache.org/jira/browse/HDFS-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HDFS-13448:
--
Assignee: BELUGA BEHR
> HDFS Block Placement - Ignore Locality
> --
>
> Key: HDFS-13448
> URL: https://issues.apache.org/jira/browse/HDFS-13448
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: block placement, hdfs-client
>Affects Versions: 2.9.0, 3.0.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-13448.1.patch
>
>
> According to the HDFS Block Place Rules:
> {quote}
> /**
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on a different node of the rack as the second replica.
> */
> {quote}
> However, there is a hint for the hdfs-client that allows the block placement
> request to not put a block replica on the local datanode _where 'local' means
> the same host as the client is being run on._
> {quote}
> /**
>* Advise that a block replica NOT be written to the local DataNode where
>* 'local' means the same host as the client is being run on.
>*
>* @see CreateFlag#NO_LOCAL_WRITE
>*/
> {quote}
> I propose that we add a new flag that allows the hdfs-client to request that
> the first block replica be placed on a random DataNode in the cluster. The
> subsequent block replicas should follow the normal block placement rules.
> The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block
> replica is not placed on the local node, but it is still placed on the local
> rack. Where this comes into play is where you have, for example, a flume
> agent that is loading data into HDFS.
> If the Flume agent is running on a DataNode, then by default, the DataNode
> local to the Flume agent will always get the first block replica and this
> leads to un-even block placements, with the local node always filling up
> faster than any other node in the cluster.
> Modifying this example, if the DataNode is removed from the host where the
> Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then
> the default block placement policy will still prefer the local rack. This
> remedies the situation only so far as now the first block replica will always
> be distributed to a DataNode on the local rack.
> This new flag would allow a single Flume agent to distribute the blocks
> randomly, evenly, over the entire cluster instead of hot-spotting the local
> node or the local rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13448) HDFS Block Placement - Ignore Locality
[
https://issues.apache.org/jira/browse/HDFS-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-13448:
---
Attachment: HDFS-13448.1.patch
> HDFS Block Placement - Ignore Locality
> --
>
> Key: HDFS-13448
> URL: https://issues.apache.org/jira/browse/HDFS-13448
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: block placement, hdfs-client
>Affects Versions: 2.9.0, 3.0.1
>Reporter: BELUGA BEHR
>Priority: Minor
> Attachments: HDFS-13448.1.patch
>
>
> According to the HDFS Block Place Rules:
> {quote}
> /**
> * The replica placement strategy is that if the writer is on a datanode,
> * the 1st replica is placed on the local machine,
> * otherwise a random datanode. The 2nd replica is placed on a datanode
> * that is on a different rack. The 3rd replica is placed on a datanode
> * which is on a different node of the rack as the second replica.
> */
> {quote}
> However, there is a hint for the hdfs-client that allows the block placement
> request to not put a block replica on the local datanode _where 'local' means
> the same host as the client is being run on._
> {quote}
> /**
>* Advise that a block replica NOT be written to the local DataNode where
>* 'local' means the same host as the client is being run on.
>*
>* @see CreateFlag#NO_LOCAL_WRITE
>*/
> {quote}
> I propose that we add a new flag that allows the hdfs-client to request that
> the first block replica be placed on a random DataNode in the cluster. The
> subsequent block replicas should follow the normal block placement rules.
> The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block
> replica is not placed on the local node, but it is still placed on the local
> rack. Where this comes into play is where you have, for example, a flume
> agent that is loading data into HDFS.
> If the Flume agent is running on a DataNode, then by default, the DataNode
> local to the Flume agent will always get the first block replica and this
> leads to un-even block placements, with the local node always filling up
> faster than any other node in the cluster.
> Modifying this example, if the DataNode is removed from the host where the
> Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then
> the default block placement policy will still prefer the local rack. This
> remedies the situation only so far as now the first block replica will always
> be distributed to a DataNode on the local rack.
> This new flag would allow a single Flume agent to distribute the blocks
> randomly, evenly, over the entire cluster instead of hot-spotting the local
> node or the local rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13442) Ozone: Handle Datanode Registration failure
[ https://issues.apache.org/jira/browse/HDFS-13442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437711#comment-16437711 ] Anu Engineer commented on HDFS-13442: - Hi [~hanishakoneru], Thanks for the patch. However, I feel that a data node should give up registration attempt after a really long time or under a condition of error. Retrying 10 times seem too low. For example, if the data nodes boot up earlier than SCM we would not want the data nodes to do silent after 10 tries (somewhere around 5 minutes) , If we are going to do a default value for max retries, we should try to target something in the order of days, say 24 hours or so. In fact, we can read the HB frequency config value and multiply that to get 24/12 hours. also in the case, we get the error, _errorNodeNotPermitted_, should we shut down the data node and create some kind of error record on SCM so we can get that info back from SCM? I am also ok with the current approach where we will let the system slowly go time out. > Ozone: Handle Datanode Registration failure > --- > > Key: HDFS-13442 > URL: https://issues.apache.org/jira/browse/HDFS-13442 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDFS-13442-HDFS-7240.001.patch > > > If a datanode is not able to register itself, we need to handle that > correctly. > If the number of unsuccessful attempts to register with the SCM exceeds a > configurable max number, the datanode should not make any more attempts. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13448) HDFS Block Placement - Ignore Locality
BELUGA BEHR created HDFS-13448:
--
Summary: HDFS Block Placement - Ignore Locality
Key: HDFS-13448
URL: https://issues.apache.org/jira/browse/HDFS-13448
Project: Hadoop HDFS
Issue Type: New Feature
Components: block placement, hdfs-client
Affects Versions: 3.0.1, 2.9.0
Reporter: BELUGA BEHR
According to the HDFS Block Place Rules:
{quote}
/**
* The replica placement strategy is that if the writer is on a datanode,
* the 1st replica is placed on the local machine,
* otherwise a random datanode. The 2nd replica is placed on a datanode
* that is on a different rack. The 3rd replica is placed on a datanode
* which is on a different node of the rack as the second replica.
*/
{quote}
However, there is a hint for the hdfs-client that allows the block placement
request to not put a block replica on the local datanode _where 'local' means
the same host as the client is being run on._
{quote}
/**
* Advise that a block replica NOT be written to the local DataNode where
* 'local' means the same host as the client is being run on.
*
* @see CreateFlag#NO_LOCAL_WRITE
*/
{quote}
I propose that we add a new flag that allows the hdfs-client to request that
the first block replica be placed on a random DataNode in the cluster. The
subsequent block replicas should follow the normal block placement rules.
The issue is that when the {{NO_LOCAL_WRITE}} is enabled, the first block
replica is not placed on the local node, but it is still placed on the local
rack. Where this comes into play is where you have, for example, a flume agent
that is loading data into HDFS.
If the Flume agent is running on a DataNode, then by default, the DataNode
local to the Flume agent will always get the first block replica and this leads
to un-even block placements, with the local node always filling up faster than
any other node in the cluster.
Modifying this example, if the DataNode is removed from the host where the
Flume agent is running, or this {{NO_LOCAL_WRITE}} is enabled by Flume, then
the default block placement policy will still prefer the local rack. This
remedies the situation only so far as now the first block replica will always
be distributed to a DataNode on the local rack.
This new flag would allow a single Flume agent to distribute the blocks
randomly, evenly, over the entire cluster instead of hot-spotting the local
node or the local rack.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13197) Ozone: Fix ConfServlet#getOzoneTags cmd after HADOOP-15007
[ https://issues.apache.org/jira/browse/HDFS-13197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13197: -- Attachment: HDFS-13197-HDFS-7240.003.patch > Ozone: Fix ConfServlet#getOzoneTags cmd after HADOOP-15007 > -- > > Key: HDFS-13197 > URL: https://issues.apache.org/jira/browse/HDFS-13197 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-13197-HDFS-7240.000.patch, > HDFS-13197-HDFS-7240.001.patch, HDFS-13197-HDFS-7240.002.patch, > HDFS-13197-HDFS-7240.003.patch, Screen Shot 2018-04-10 at 2.05.35 PM.png > > > This is broken after merging trunk change HADOOP-15007 into HDFS-7240 branch. > I remove the cmd and related test to have a clean merge. [~ajakumar], please > fix the cmd and bring back the related test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13197) Ozone: Fix ConfServlet#getOzoneTags cmd after HADOOP-15007
[ https://issues.apache.org/jira/browse/HDFS-13197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437671#comment-16437671 ] Ajay Kumar commented on HDFS-13197: --- [~xyao], Thanks for review. Addressed both comments in patch v3. > Ozone: Fix ConfServlet#getOzoneTags cmd after HADOOP-15007 > -- > > Key: HDFS-13197 > URL: https://issues.apache.org/jira/browse/HDFS-13197 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-13197-HDFS-7240.000.patch, > HDFS-13197-HDFS-7240.001.patch, HDFS-13197-HDFS-7240.002.patch, Screen Shot > 2018-04-10 at 2.05.35 PM.png > > > This is broken after merging trunk change HADOOP-15007 into HDFS-7240 branch. > I remove the cmd and related test to have a clean merge. [~ajakumar], please > fix the cmd and bring back the related test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13330) ShortCircuitCache#fetchOrCreate never retries
[
https://issues.apache.org/jira/browse/HDFS-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437658#comment-16437658
]
genericqa commented on HDFS-13330:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 1m
7s{color} | {color:red} Docker failed to build yetus/hadoop:f667ef1. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-13330 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918984/HDFS-13330.branch-2.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23930/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ShortCircuitCache#fetchOrCreate never retries
> -
>
> Key: HDFS-13330
> URL: https://issues.apache.org/jira/browse/HDFS-13330
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HDFS-13330.001.patch, HDFS-13330.002.patch,
> HDFS-13330.004.patch, HDFS-13330.005.patch, HDFS-13330.branch-2.patch
>
>
> The follow do .. while(false) loop seems useless to me. The code intended to
> retry but it never worked. Let's fix it.
> {code:java:title=ShortCircuitCache#fetchOrCreate}
> ShortCircuitReplicaInfo info = null;
> do {
> if (closed) {
> LOG.trace("{}: can't fethchOrCreate {} because the cache is closed.",
> this, key);
> return null;
> }
> Waitable waitable = replicaInfoMap.get(key);
> if (waitable != null) {
> try {
> info = fetch(key, waitable);
> } catch (RetriableException e) {
> LOG.debug("{}: retrying {}", this, e.getMessage());
> }
> }
> } while (false);{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13315) Add a test for the issue reported in HDFS-11481 which is fixed by HDFS-10997.
[ https://issues.apache.org/jira/browse/HDFS-13315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437656#comment-16437656 ] Brahma Reddy Battula commented on HDFS-13315: - Thanks for working on this, this needs to be marked as resolved..? > Add a test for the issue reported in HDFS-11481 which is fixed by HDFS-10997. > - > > Key: HDFS-13315 > URL: https://issues.apache.org/jira/browse/HDFS-13315 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Major > Fix For: 3.1.1 > > Attachments: HDFS-13315.001.patch, HDFS-13315.002.patch, > TEST-org.apache.hadoop.hdfs.TestEncryptionZones.xml > > > HDFS-11481 reported that hdfs snapshotDiff /.reserved/raw/... fails on > snapshottable directories. It turns out that HDFS-10997 fixed the issue as a > byproduct. This jira is to add a test for the HDFS-11481 issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10997) Reduce number of path resolving methods
[ https://issues.apache.org/jira/browse/HDFS-10997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437634#comment-16437634 ] Arpit Agarwal edited comment on HDFS-10997 at 4/13/18 5:43 PM: --- -Cherry-picked to branch-3.1.- (Ignore this, erroneously included in a bulk edit). was (Author: arpitagarwal): Cherry-picked to branch-3.1. > Reduce number of path resolving methods > --- > > Key: HDFS-10997 > URL: https://issues.apache.org/jira/browse/HDFS-10997 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Major > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-10997.1.patch, HDFS-10997.2.patch, > HDFS-10997.branch-2.patch, HDFS-10997.patch > > > FSDirectory contains many methods for resolving paths to an IIP and/or inode. > These should be unified into a couple methods that will consistently do the > basics of resolving reserved paths, blocking write ops from snapshot paths, > verifying ancestors as directories, and throwing if symlinks are encountered. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10997) Reduce number of path resolving methods
[ https://issues.apache.org/jira/browse/HDFS-10997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10997: - Fix Version/s: (was: 3.1.1) > Reduce number of path resolving methods > --- > > Key: HDFS-10997 > URL: https://issues.apache.org/jira/browse/HDFS-10997 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Major > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-10997.1.patch, HDFS-10997.2.patch, > HDFS-10997.branch-2.patch, HDFS-10997.patch > > > FSDirectory contains many methods for resolving paths to an IIP and/or inode. > These should be unified into a couple methods that will consistently do the > basics of resolving reserved paths, blocking write ops from snapshot paths, > verifying ancestors as directories, and throwing if symlinks are encountered. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13330) ShortCircuitCache#fetchOrCreate never retries
[
https://issues.apache.org/jira/browse/HDFS-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437644#comment-16437644
]
Wei-Chiu Chuang commented on HDFS-13330:
Attach a branch-2 patch. There's a very trivial change in the test code.
{code:java}
.fetch(Mockito.eq(extendedBlockId), Mockito.any(Waitable.class));
{code}
> ShortCircuitCache#fetchOrCreate never retries
> -
>
> Key: HDFS-13330
> URL: https://issues.apache.org/jira/browse/HDFS-13330
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HDFS-13330.001.patch, HDFS-13330.002.patch,
> HDFS-13330.004.patch, HDFS-13330.005.patch, HDFS-13330.branch-2.patch
>
>
> The follow do .. while(false) loop seems useless to me. The code intended to
> retry but it never worked. Let's fix it.
> {code:java:title=ShortCircuitCache#fetchOrCreate}
> ShortCircuitReplicaInfo info = null;
> do {
> if (closed) {
> LOG.trace("{}: can't fethchOrCreate {} because the cache is closed.",
> this, key);
> return null;
> }
> Waitable waitable = replicaInfoMap.get(key);
> if (waitable != null) {
> try {
> info = fetch(key, waitable);
> } catch (RetriableException e) {
> LOG.debug("{}: retrying {}", this, e.getMessage());
> }
> }
> } while (false);{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13330) ShortCircuitCache#fetchOrCreate never retries
[
https://issues.apache.org/jira/browse/HDFS-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-13330:
---
Attachment: HDFS-13330.branch-2.patch
> ShortCircuitCache#fetchOrCreate never retries
> -
>
> Key: HDFS-13330
> URL: https://issues.apache.org/jira/browse/HDFS-13330
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HDFS-13330.001.patch, HDFS-13330.002.patch,
> HDFS-13330.004.patch, HDFS-13330.005.patch, HDFS-13330.branch-2.patch
>
>
> The follow do .. while(false) loop seems useless to me. The code intended to
> retry but it never worked. Let's fix it.
> {code:java:title=ShortCircuitCache#fetchOrCreate}
> ShortCircuitReplicaInfo info = null;
> do {
> if (closed) {
> LOG.trace("{}: can't fethchOrCreate {} because the cache is closed.",
> this, key);
> return null;
> }
> Waitable waitable = replicaInfoMap.get(key);
> if (waitable != null) {
> try {
> info = fetch(key, waitable);
> } catch (RetriableException e) {
> LOG.debug("{}: retrying {}", this, e.getMessage());
> }
> }
> } while (false);{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11043) TestWebHdfsTimeouts fails
[ https://issues.apache.org/jira/browse/HDFS-11043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-11043: - Fix Version/s: 3.1.1 Cherry-picked to branch-3.1. > TestWebHdfsTimeouts fails > - > > Key: HDFS-11043 > URL: https://issues.apache.org/jira/browse/HDFS-11043 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Chao Sun >Priority: Major > Fix For: 3.2.0, 3.1.1 > > Attachments: HDFS-11043.000.patch, HDFS-11043.001.patch, > HDFS-11043.002.patch, org.apache.hadoop.hdfs.web.TestWebHdfsTimeouts.txt > > > I'm seeing reproducible test failures for TestWebHdfsTimeouts locally, at > least on trunk. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13315) Add a test for the issue reported in HDFS-11481 which is fixed by HDFS-10997.
[ https://issues.apache.org/jira/browse/HDFS-13315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-13315: - Fix Version/s: 3.1.1 Cherry-picked to branch-3.1. > Add a test for the issue reported in HDFS-11481 which is fixed by HDFS-10997. > - > > Key: HDFS-13315 > URL: https://issues.apache.org/jira/browse/HDFS-13315 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Major > Fix For: 3.1.1 > > Attachments: HDFS-13315.001.patch, HDFS-13315.002.patch, > TEST-org.apache.hadoop.hdfs.TestEncryptionZones.xml > > > HDFS-11481 reported that hdfs snapshotDiff /.reserved/raw/... fails on > snapshottable directories. It turns out that HDFS-10997 fixed the issue as a > byproduct. This jira is to add a test for the HDFS-11481 issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10997) Reduce number of path resolving methods
[ https://issues.apache.org/jira/browse/HDFS-10997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-10997: - Fix Version/s: 3.1.1 Cherry-picked to branch-3.1. > Reduce number of path resolving methods > --- > > Key: HDFS-10997 > URL: https://issues.apache.org/jira/browse/HDFS-10997 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Major > Fix For: 2.8.0, 3.0.0-alpha2, 3.1.1 > > Attachments: HDFS-10997.1.patch, HDFS-10997.2.patch, > HDFS-10997.branch-2.patch, HDFS-10997.patch > > > FSDirectory contains many methods for resolving paths to an IIP and/or inode. > These should be unified into a couple methods that will consistently do the > basics of resolving reserved paths, blocking write ops from snapshot paths, > verifying ancestors as directories, and throwing if symlinks are encountered. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11900) Hedged reads thread pool creation not synchronized
[
https://issues.apache.org/jira/browse/HDFS-11900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-11900:
-
Fix Version/s: 3.1.1
Cherry-picked to branch-3.1.
> Hedged reads thread pool creation not synchronized
> --
>
> Key: HDFS-11900
> URL: https://issues.apache.org/jira/browse/HDFS-11900
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: HDFS-11900.001.patch
>
>
> *Non-static* synchronized method initThreadsNumForHedgedReads can't
> synchronize the access to the *static* class variable HEDGED_READ_THREAD_POOL.
> {code}
> private static ThreadPoolExecutor HEDGED_READ_THREAD_POOL;
> ...
> private synchronized void initThreadsNumForHedgedReads(int num) {
> {code}
> 2 DFS clients may update the same static variable in a race because the lock
> is on each DFS client object, not on the shared DFSClient class object.
> There are 2 possible fixes:
> 1. "Global thread pool": Change initThreadsNumForHedgedReads to static
> 2. "Per-client thread pool": Change HEDGED_READ_THREAD_POOL to non-static
> From the description for property {{dfs.client.hedged.read.threadpool.size}}:
> {quote}
> to a positive number. The threadpool size is how many threads to dedicate
> to the running of these 'hedged', concurrent reads in your client.
> {quote}
> it seems to indicate the thread pool is per DFS client.
> Let's assume we go with #1 "Global thread pool". One DFS client has the
> property set to 10 in its config, while the other client has the property set
> to 5 in its config, what is supposed to the size of the global thread pool?
> 5? 10? Or 15?
> The 2nd fix seems more reasonable to me.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12828) OIV ReverseXML Processor Fails With Escaped Characters
[
https://issues.apache.org/jira/browse/HDFS-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437612#comment-16437612
]
Erik Krogen commented on HDFS-12828:
The issue comes from how the {{XMLEventReader}} processing entity references.
The code assumed that a XML block like {{foobar}} would be
parsed as a {{START_ELEMENT}}, a {{CHARACTERS}} with "foobar", and an
{{END_ELEMENT}}. However what actually happens between the start/end element is
three {{CHARACTERS}} blocks, "foo", "&", and "bar" (note that the entity
reference has already been handled). So, remove the flag to process entity
references, and support multiple contiguous {{CHARACTERS}} blocks.
Attached a patch with the fix and supplementing existing unit tests.
> OIV ReverseXML Processor Fails With Escaped Characters
> --
>
> Key: HDFS-12828
> URL: https://issues.apache.org/jira/browse/HDFS-12828
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-12828.000.patch, fsimage_008.xml
>
>
> The HDFS OIV ReverseXML processor fails if the XML file contains escaped
> characters:
> {code}
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs dfs -fs hdfs://localhost:9000/ -ls /
> Found 4 items
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:48 /foo
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:49 /foo"
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:50 /foo`
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:49 /foo&
> {code}
> Then after doing {{saveNamespace}} on that NameNode...
> {code}
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs oiv -i
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008 -o
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml -p XML
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs oiv -i
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml -o
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml.rev -p
> ReverseXML
> OfflineImageReconstructor failed: unterminated entity ref starting with &
> org.apache.hadoop.hdfs.util.XMLUtils$UnmanglingError: unterminated entity ref
> starting with &
> at
> org.apache.hadoop.hdfs.util.XMLUtils.unmangleXmlString(XMLUtils.java:232)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildrenHelper(OfflineImageReconstructor.java:383)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildrenHelper(OfflineImageReconstructor.java:379)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildren(OfflineImageReconstructor.java:418)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.access$1000(OfflineImageReconstructor.java:95)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$INodeSectionProcessor.process(OfflineImageReconstructor.java:524)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1710)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1765)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:191)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:134)
> {code}
> See attachments for relevant fsimage XML file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12828) OIV ReverseXML Processor Fails With Escaped Characters
[
https://issues.apache.org/jira/browse/HDFS-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-12828:
---
Attachment: (was: HDFS-12818.000.patch)
> OIV ReverseXML Processor Fails With Escaped Characters
> --
>
> Key: HDFS-12828
> URL: https://issues.apache.org/jira/browse/HDFS-12828
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-12828.000.patch, fsimage_008.xml
>
>
> The HDFS OIV ReverseXML processor fails if the XML file contains escaped
> characters:
> {code}
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs dfs -fs hdfs://localhost:9000/ -ls /
> Found 4 items
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:48 /foo
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:49 /foo"
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:50 /foo`
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:49 /foo&
> {code}
> Then after doing {{saveNamespace}} on that NameNode...
> {code}
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs oiv -i
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008 -o
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml -p XML
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs oiv -i
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml -o
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml.rev -p
> ReverseXML
> OfflineImageReconstructor failed: unterminated entity ref starting with &
> org.apache.hadoop.hdfs.util.XMLUtils$UnmanglingError: unterminated entity ref
> starting with &
> at
> org.apache.hadoop.hdfs.util.XMLUtils.unmangleXmlString(XMLUtils.java:232)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildrenHelper(OfflineImageReconstructor.java:383)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildrenHelper(OfflineImageReconstructor.java:379)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildren(OfflineImageReconstructor.java:418)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.access$1000(OfflineImageReconstructor.java:95)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$INodeSectionProcessor.process(OfflineImageReconstructor.java:524)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1710)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1765)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:191)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:134)
> {code}
> See attachments for relevant fsimage XML file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12828) OIV ReverseXML Processor Fails With Escaped Characters
[
https://issues.apache.org/jira/browse/HDFS-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-12828:
---
Attachment: HDFS-12818.000.patch
> OIV ReverseXML Processor Fails With Escaped Characters
> --
>
> Key: HDFS-12828
> URL: https://issues.apache.org/jira/browse/HDFS-12828
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-12828.000.patch, fsimage_008.xml
>
>
> The HDFS OIV ReverseXML processor fails if the XML file contains escaped
> characters:
> {code}
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs dfs -fs hdfs://localhost:9000/ -ls /
> Found 4 items
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:48 /foo
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:49 /foo"
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:50 /foo`
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:49 /foo&
> {code}
> Then after doing {{saveNamespace}} on that NameNode...
> {code}
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs oiv -i
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008 -o
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml -p XML
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs oiv -i
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml -o
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml.rev -p
> ReverseXML
> OfflineImageReconstructor failed: unterminated entity ref starting with &
> org.apache.hadoop.hdfs.util.XMLUtils$UnmanglingError: unterminated entity ref
> starting with &
> at
> org.apache.hadoop.hdfs.util.XMLUtils.unmangleXmlString(XMLUtils.java:232)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildrenHelper(OfflineImageReconstructor.java:383)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildrenHelper(OfflineImageReconstructor.java:379)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildren(OfflineImageReconstructor.java:418)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.access$1000(OfflineImageReconstructor.java:95)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$INodeSectionProcessor.process(OfflineImageReconstructor.java:524)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1710)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1765)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:191)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:134)
> {code}
> See attachments for relevant fsimage XML file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12828) OIV ReverseXML Processor Fails With Escaped Characters
[
https://issues.apache.org/jira/browse/HDFS-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-12828:
---
Attachment: HDFS-12828.000.patch
> OIV ReverseXML Processor Fails With Escaped Characters
> --
>
> Key: HDFS-12828
> URL: https://issues.apache.org/jira/browse/HDFS-12828
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-12828.000.patch, fsimage_008.xml
>
>
> The HDFS OIV ReverseXML processor fails if the XML file contains escaped
> characters:
> {code}
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs dfs -fs hdfs://localhost:9000/ -ls /
> Found 4 items
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:48 /foo
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:49 /foo"
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:50 /foo`
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:49 /foo&
> {code}
> Then after doing {{saveNamespace}} on that NameNode...
> {code}
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs oiv -i
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008 -o
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml -p XML
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs oiv -i
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml -o
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml.rev -p
> ReverseXML
> OfflineImageReconstructor failed: unterminated entity ref starting with &
> org.apache.hadoop.hdfs.util.XMLUtils$UnmanglingError: unterminated entity ref
> starting with &
> at
> org.apache.hadoop.hdfs.util.XMLUtils.unmangleXmlString(XMLUtils.java:232)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildrenHelper(OfflineImageReconstructor.java:383)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildrenHelper(OfflineImageReconstructor.java:379)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildren(OfflineImageReconstructor.java:418)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.access$1000(OfflineImageReconstructor.java:95)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$INodeSectionProcessor.process(OfflineImageReconstructor.java:524)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1710)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1765)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:191)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:134)
> {code}
> See attachments for relevant fsimage XML file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12828) OIV ReverseXML Processor Fails With Escaped Characters
[
https://issues.apache.org/jira/browse/HDFS-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-12828:
---
Status: Patch Available (was: In Progress)
> OIV ReverseXML Processor Fails With Escaped Characters
> --
>
> Key: HDFS-12828
> URL: https://issues.apache.org/jira/browse/HDFS-12828
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: fsimage_008.xml
>
>
> The HDFS OIV ReverseXML processor fails if the XML file contains escaped
> characters:
> {code}
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs dfs -fs hdfs://localhost:9000/ -ls /
> Found 4 items
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:48 /foo
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:49 /foo"
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:50 /foo`
> drwxr-xr-x - ekrogen supergroup 0 2017-11-16 14:49 /foo&
> {code}
> Then after doing {{saveNamespace}} on that NameNode...
> {code}
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs oiv -i
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008 -o
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml -p XML
> ekrogen at ekrogen-ld1 in
> ~/dev/hadoop/trunk/hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT on trunk!
> ± $HADOOP_HOME/bin/hdfs oiv -i
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml -o
> /tmp/hadoop-ekrogen/dfs/name/current/fsimage_008.xml.rev -p
> ReverseXML
> OfflineImageReconstructor failed: unterminated entity ref starting with &
> org.apache.hadoop.hdfs.util.XMLUtils$UnmanglingError: unterminated entity ref
> starting with &
> at
> org.apache.hadoop.hdfs.util.XMLUtils.unmangleXmlString(XMLUtils.java:232)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildrenHelper(OfflineImageReconstructor.java:383)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildrenHelper(OfflineImageReconstructor.java:379)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.loadNodeChildren(OfflineImageReconstructor.java:418)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.access$1000(OfflineImageReconstructor.java:95)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$INodeSectionProcessor.process(OfflineImageReconstructor.java:524)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1710)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1765)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:191)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:134)
> {code}
> See attachments for relevant fsimage XML file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13447) Fix Typos - Node Not Chosen
BELUGA BEHR created HDFS-13447:
--
Summary: Fix Typos - Node Not Chosen
Key: HDFS-13447
URL: https://issues.apache.org/jira/browse/HDFS-13447
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.0.1, 2.2.0
Reporter: BELUGA BEHR
Attachments: HDFS-13447.1.patch
Fix typo and improve:
{code:java}
private enum NodeNotChosenReason {
NOT_IN_SERVICE("the node isn't in service"),
NODE_STALE("the node is stale"),
NODE_TOO_BUSY("the node is too busy"),
TOO_MANY_NODES_ON_RACK("the rack has too many chosen nodes"),
NOT_ENOUGH_STORAGE_SPACE("no enough storage space to place the block");{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13447) Fix Typos - Node Not Chosen
[
https://issues.apache.org/jira/browse/HDFS-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-13447:
---
Status: Patch Available (was: Open)
> Fix Typos - Node Not Chosen
> ---
>
> Key: HDFS-13447
> URL: https://issues.apache.org/jira/browse/HDFS-13447
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.1, 2.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HDFS-13447.1.patch
>
>
> Fix typo and improve:
>
> {code:java}
> private enum NodeNotChosenReason {
> NOT_IN_SERVICE("the node isn't in service"),
> NODE_STALE("the node is stale"),
> NODE_TOO_BUSY("the node is too busy"),
> TOO_MANY_NODES_ON_RACK("the rack has too many chosen nodes"),
> NOT_ENOUGH_STORAGE_SPACE("no enough storage space to place the
> block");{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13447) Fix Typos - Node Not Chosen
[
https://issues.apache.org/jira/browse/HDFS-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HDFS-13447:
--
Assignee: BELUGA BEHR
> Fix Typos - Node Not Chosen
> ---
>
> Key: HDFS-13447
> URL: https://issues.apache.org/jira/browse/HDFS-13447
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.2.0, 3.0.1
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HDFS-13447.1.patch
>
>
> Fix typo and improve:
>
> {code:java}
> private enum NodeNotChosenReason {
> NOT_IN_SERVICE("the node isn't in service"),
> NODE_STALE("the node is stale"),
> NODE_TOO_BUSY("the node is too busy"),
> TOO_MANY_NODES_ON_RACK("the rack has too many chosen nodes"),
> NOT_ENOUGH_STORAGE_SPACE("no enough storage space to place the
> block");{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13447) Fix Typos - Node Not Chosen
[
https://issues.apache.org/jira/browse/HDFS-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HDFS-13447:
---
Attachment: HDFS-13447.1.patch
> Fix Typos - Node Not Chosen
> ---
>
> Key: HDFS-13447
> URL: https://issues.apache.org/jira/browse/HDFS-13447
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.2.0, 3.0.1
>Reporter: BELUGA BEHR
>Priority: Trivial
> Attachments: HDFS-13447.1.patch
>
>
> Fix typo and improve:
>
> {code:java}
> private enum NodeNotChosenReason {
> NOT_IN_SERVICE("the node isn't in service"),
> NODE_STALE("the node is stale"),
> NODE_TOO_BUSY("the node is too busy"),
> TOO_MANY_NODES_ON_RACK("the rack has too many chosen nodes"),
> NOT_ENOUGH_STORAGE_SPACE("no enough storage space to place the
> block");{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13446) Ozone: Fix OzoneFileSystem contract test failures
[
https://issues.apache.org/jira/browse/HDFS-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437569#comment-16437569
]
genericqa commented on HDFS-13446:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 18 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
24s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
28s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
26s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
26s{color} | {color:red} hadoop-ozone in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} hadoop-ozone in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
25s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
24s{color} | {color:red} hadoop-ozone in HDFS-7240 failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
10s{color} | {color:red} container-service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
10s{color} | {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
10s{color} | {color:red} client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
12s{color} | {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} container-service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} server-scm in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
20s{color} | {color:red} client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
30s{color}
[jira] [Commented] (HDFS-13330) ShortCircuitCache#fetchOrCreate never retries
[
https://issues.apache.org/jira/browse/HDFS-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437560#comment-16437560
]
Hudson commented on HDFS-13330:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13994 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13994/])
HDFS-13330. ShortCircuitCache#fetchOrCreate never retries. Contributed
(weichiu: rev e66e287efe2b43e710137a628f03c7df3ebdf498)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitCache.java
> ShortCircuitCache#fetchOrCreate never retries
> -
>
> Key: HDFS-13330
> URL: https://issues.apache.org/jira/browse/HDFS-13330
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HDFS-13330.001.patch, HDFS-13330.002.patch,
> HDFS-13330.004.patch, HDFS-13330.005.patch
>
>
> The follow do .. while(false) loop seems useless to me. The code intended to
> retry but it never worked. Let's fix it.
> {code:java:title=ShortCircuitCache#fetchOrCreate}
> ShortCircuitReplicaInfo info = null;
> do {
> if (closed) {
> LOG.trace("{}: can't fethchOrCreate {} because the cache is closed.",
> this, key);
> return null;
> }
> Waitable waitable = replicaInfoMap.get(key);
> if (waitable != null) {
> try {
> info = fetch(key, waitable);
> } catch (RetriableException e) {
> LOG.debug("{}: retrying {}", this, e.getMessage());
> }
> }
> } while (false);{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13438) Fix javadoc in FsVolumeList#removeVolume
[ https://issues.apache.org/jira/browse/HDFS-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437523#comment-16437523 ] Hudson commented on HDFS-13438: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13993 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13993/]) HDFS-13438. Fix javadoc in FsVolumeList#removeVolume. Contributed by (Bharat: rev fa8b88ab2b272b29cf116a5de038d78fc4357b9d) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java > Fix javadoc in FsVolumeList#removeVolume > > > Key: HDFS-13438 > URL: https://issues.apache.org/jira/browse/HDFS-13438 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Fix For: 3.2.0, 3.1.1 > > Attachments: HDFS-13438.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13430) Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445
[ https://issues.apache.org/jira/browse/HDFS-13430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437524#comment-16437524 ] Hudson commented on HDFS-13430: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13993 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13993/]) HDFS-13430. Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445. (xiao: rev 650359371175fba416331e73aa03d2a96ccb90e5) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java > Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445 > -- > > Key: HDFS-13430 > URL: https://issues.apache.org/jira/browse/HDFS-13430 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Fix For: 3.2.0 > > Attachments: HDFS-13430.01.patch > > > Unfortunately HADOOP-14445 had an HDFS test failure that's not caught in the > hadoop-common precommit runs. > This is caught by our internal pre-commit using dist-test, and appears to be > the only failure. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13330) ShortCircuitCache#fetchOrCreate never retries
[
https://issues.apache.org/jira/browse/HDFS-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-13330:
---
Fix Version/s: 3.2.0
> ShortCircuitCache#fetchOrCreate never retries
> -
>
> Key: HDFS-13330
> URL: https://issues.apache.org/jira/browse/HDFS-13330
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HDFS-13330.001.patch, HDFS-13330.002.patch,
> HDFS-13330.004.patch, HDFS-13330.005.patch
>
>
> The follow do .. while(false) loop seems useless to me. The code intended to
> retry but it never worked. Let's fix it.
> {code:java:title=ShortCircuitCache#fetchOrCreate}
> ShortCircuitReplicaInfo info = null;
> do {
> if (closed) {
> LOG.trace("{}: can't fethchOrCreate {} because the cache is closed.",
> this, key);
> return null;
> }
> Waitable waitable = replicaInfoMap.get(key);
> if (waitable != null) {
> try {
> info = fetch(key, waitable);
> } catch (RetriableException e) {
> LOG.debug("{}: retrying {}", this, e.getMessage());
> }
> }
> } while (false);{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13438) Fix javadoc in FsVolumeList#removeVolume
[ https://issues.apache.org/jira/browse/HDFS-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13438: -- Hadoop Flags: Reviewed > Fix javadoc in FsVolumeList#removeVolume > > > Key: HDFS-13438 > URL: https://issues.apache.org/jira/browse/HDFS-13438 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Fix For: 3.2.0, 3.1.1 > > Attachments: HDFS-13438.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13438) Fix javadoc in FsVolumeList#removeVolume
[ https://issues.apache.org/jira/browse/HDFS-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437481#comment-16437481 ] Bharat Viswanadham edited comment on HDFS-13438 at 4/13/18 4:07 PM: Thank You [~shashikant] for reporting and working on this. I have committed to trunk and branch-3.1 was (Author: bharatviswa): Thank You [~shashikant] for reporting and working on this. I have committed to trunk. > Fix javadoc in FsVolumeList#removeVolume > > > Key: HDFS-13438 > URL: https://issues.apache.org/jira/browse/HDFS-13438 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Fix For: 3.2.0, 3.1.1 > > Attachments: HDFS-13438.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13438) Fix javadoc in FsVolumeList#removeVolume
[ https://issues.apache.org/jira/browse/HDFS-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13438: -- Fix Version/s: 3.1.1 > Fix javadoc in FsVolumeList#removeVolume > > > Key: HDFS-13438 > URL: https://issues.apache.org/jira/browse/HDFS-13438 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Fix For: 3.2.0, 3.1.1 > > Attachments: HDFS-13438.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13430) Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445
[ https://issues.apache.org/jira/browse/HDFS-13430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437488#comment-16437488 ] Xiao Chen commented on HDFS-13430: -- Thanks Rushabh. Committed to trunk, will let this open for backports. > Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445 > -- > > Key: HDFS-13430 > URL: https://issues.apache.org/jira/browse/HDFS-13430 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Fix For: 3.2.0 > > Attachments: HDFS-13430.01.patch > > > Unfortunately HADOOP-14445 had an HDFS test failure that's not caught in the > hadoop-common precommit runs. > This is caught by our internal pre-commit using dist-test, and appears to be > the only failure. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13430) Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445
[ https://issues.apache.org/jira/browse/HDFS-13430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-13430: - Fix Version/s: 3.2.0 > Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445 > -- > > Key: HDFS-13430 > URL: https://issues.apache.org/jira/browse/HDFS-13430 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Fix For: 3.2.0 > > Attachments: HDFS-13430.01.patch > > > Unfortunately HADOOP-14445 had an HDFS test failure that's not caught in the > hadoop-common precommit runs. > This is caught by our internal pre-commit using dist-test, and appears to be > the only failure. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13435) RBF: Improve the error loggings for printing the stack trace
[
https://issues.apache.org/jira/browse/HDFS-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437485#comment-16437485
]
Íñigo Goiri edited comment on HDFS-13435 at 4/13/18 4:02 PM:
-
Not sure I would add the full stack trace for a couple of those:
* FederationMetrics#268, I think that the message should enough in this case to
know if it's the JSON or the network that fails.
* RouterRpcServer#2373, I think the message should be enough.
* StateStoreZooKeeperImpl usually gets pretty decent messages from curator.
Maybe we could extend the tests to trigger more of these errors.
I think the main issue at this point is that we use IOException for everything
and it's hard to tell which one is what.
was (Author: elgoiri):
Not sure I would add the full stack trace for a couple of those:
* FederationMetrics#268, I think that the message should enough in this case to
know if it's the JSON or the network that fails.
* RouterRpcServer#2373, I think the message should be enough.
* StateStoreZooKeeperImpl usually gets pretty decent messages from curator.
Maybe we could extend the tests to trigger more of these errors.
I think the main issue at this point is that we use IOException for everything
and it's hard to tell which one is what.
> RBF: Improve the error loggings for printing the stack trace
>
>
> Key: HDFS-13435
> URL: https://issues.apache.org/jira/browse/HDFS-13435
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-13435.001.patch, HDFS-13435.002.patch
>
>
> There are many places that using {{Logger.error(String format, Object...
> arguments)}} incorrectly.
> A example:
> {code:java}
> LOG.error("Cannot remove {}", path, e);
> {code}
> The exception passed here is no meaning and won't be printed. Actually it
> should be update to
> {code:java}
> LOG.error("Cannot remove {}: {}.", path, e.getMessage());
> {code}
> or
> {code:java}
> LOG.error("Cannot remove " + path, e));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13435) RBF: Improve the error loggings for printing the stack trace
[
https://issues.apache.org/jira/browse/HDFS-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437485#comment-16437485
]
Íñigo Goiri commented on HDFS-13435:
Not sure I would add the full stack trace for a couple of those:
* FederationMetrics#268, I think that the message should enough in this case to
know if it's the JSON or the network that fails.
* RouterRpcServer#2373, I think the message should be enough.
* StateStoreZooKeeperImpl usually gets pretty decent messages from curator.
Maybe we could extend the tests to trigger more of these errors.
I think the main issue at this point is that we use IOException for everything
and it's hard to tell which one is what.
> RBF: Improve the error loggings for printing the stack trace
>
>
> Key: HDFS-13435
> URL: https://issues.apache.org/jira/browse/HDFS-13435
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-13435.001.patch, HDFS-13435.002.patch
>
>
> There are many places that using {{Logger.error(String format, Object...
> arguments)}} incorrectly.
> A example:
> {code:java}
> LOG.error("Cannot remove {}", path, e);
> {code}
> The exception passed here is no meaning and won't be printed. Actually it
> should be update to
> {code:java}
> LOG.error("Cannot remove {}: {}.", path, e.getMessage());
> {code}
> or
> {code:java}
> LOG.error("Cannot remove " + path, e));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13438) Fix javadoc in FsVolumeList#removeVolume
[ https://issues.apache.org/jira/browse/HDFS-13438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13438: -- Resolution: Fixed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) Thank You [~shashikant] for reporting and working on this. I have committed to trunk. > Fix javadoc in FsVolumeList#removeVolume > > > Key: HDFS-13438 > URL: https://issues.apache.org/jira/browse/HDFS-13438 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-13438.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13388) RequestHedgingProxyProvider calls multiple configured NNs all the time
[
https://issues.apache.org/jira/browse/HDFS-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437468#comment-16437468
]
Íñigo Goiri commented on HDFS-13388:
Yetus build is clean and it looks like the test run
[succesfully|https://builds.apache.org/job/PreCommit-HDFS-Build/23918/testReport/org.apache.hadoop.hdfs.server.namenode.ha/TestRequestHedgingProxyProvider/].
Minor nits:
* static import for {{GenericTestUtils.assertExceptionContains()}}.
* We could use a different counter for the bad and good to track this more
accurately.
* Add some comments explaining the purpose and the expectation of the two new
tests.
> RequestHedgingProxyProvider calls multiple configured NNs all the time
> --
>
> Key: HDFS-13388
> URL: https://issues.apache.org/jira/browse/HDFS-13388
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HADOOP-13388.0001.patch, HADOOP-13388.0002.patch,
> HADOOP-13388.0003.patch, HADOOP-13388.0004.patch, HADOOP-13388.0005.patch,
> HADOOP-13388.0006.patch, HADOOP-13388.0007.patch, HADOOP-13388.0008.patch,
> HADOOP-13388.0009.patch, HADOOP-13388.0010.patch, HADOOP-13388.0011.patch,
> HADOOP-13388.0012.patch
>
>
> In HDFS-7858 RequestHedgingProxyProvider was designed to "first
> simultaneously call multiple configured NNs to decide which is the active
> Namenode and then for subsequent calls it will invoke the previously
> successful NN ." But the current code call multiple configured NNs every time
> even when we already got the successful NN.
> That's because in RetryInvocationHandler.java, ProxyDescriptor's member
> proxyInfo is assigned only when it is constructed or when failover occurs.
> RequestHedgingProxyProvider.currentUsedProxy is null in both cases, so the
> only proxy we can get is always a dynamic proxy handled by
> RequestHedgingInvocationHandler.class. RequestHedgingInvocationHandler.class
> handles invoked method by calling multiple configured NNs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[
https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437465#comment-16437465
]
Íñigo Goiri commented on HDFS-13443:
Thanks [~arshad.mohammad] for reporting.
The ones related might be HDFS-13212 and 13233.
For the Rebalancer (still WIP in HDFS-13123), we use something similar but
without pushing to the Routers.
The way is done (still internal and not posted) there is by checking that all
the Routers have the desired version (this comes from {{RouterStore }}) of the
Mount Table.
Then, the Rebalancer waits until this happens.
CC [~ywskycn].
> RBF: Update mount table cache immediately after changing (add/update/remove)
> mount table entries.
> -
>
> Key: HDFS-13443
> URL: https://issues.apache.org/jira/browse/HDFS-13443
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Mohammad Arshad
>Priority: Major
> Labels: RBF
>
> Currently mount table cache is updated periodically, by default cache is
> updated every minute. After change in mount table, user operations may still
> use old mount table. This is bit wrong.
> To update mount table cache, maybe we can do following
> * *Add refresh API in MountTableManager which will update mount table cache.*
> * *When there is a change in mount table entries, router admin server can
> update its cache and ask other routers to update their cache*. For example if
> there are three routers R1,R2,R3 in a cluster then add mount table entry API,
> at admin server side, will perform following sequence of action
> ## user submit add mount table entry request on R1
> ## R1 adds the mount table entry in state store
> ## R1 call refresh API on R2
> ## R1 calls refresh API on R3
> ## R1 directly freshest its cache
> ## Add mount table entry response send back to user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[ https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13443: --- Summary: RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries. (was: Update mount table cache immediately after changing (add/update/remove) mount table entries.) > RBF: Update mount table cache immediately after changing (add/update/remove) > mount table entries. > - > > Key: HDFS-13443 > URL: https://issues.apache.org/jira/browse/HDFS-13443 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Mohammad Arshad >Priority: Major > Labels: RBF > > Currently mount table cache is updated periodically, by default cache is > updated every minute. After change in mount table, user operations may still > use old mount table. This is bit wrong. > To update mount table cache, maybe we can do following > * *Add refresh API in MountTableManager which will update mount table cache.* > * *When there is a change in mount table entries, router admin server can > update its cache and ask other routers to update their cache*. For example if > there are three routers R1,R2,R3 in a cluster then add mount table entry API, > at admin server side, will perform following sequence of action > ## user submit add mount table entry request on R1 > ## R1 adds the mount table entry in state store > ## R1 call refresh API on R2 > ## R1 calls refresh API on R3 > ## R1 directly freshest its cache > ## Add mount table entry response send back to user. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13381) [SPS]: Use DFSUtilClient#makePathFromFileId() to prepare satisfier file path
[
https://issues.apache.org/jira/browse/HDFS-13381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437443#comment-16437443
]
Rakesh R commented on HDFS-13381:
-
Thanks a lot [~daryn] for the comments. Kindly, go through my reply.
Please feel free to correct me if I misunderstood any of your thoughts:)
{quote}The overall intent of using file id paths is completely decoupling the
file id collector from the namesystem
{quote}
Adding few details about the SPS design change proposal in this jira. As we
know, {{Context}} has two specific implementations - (1){{ExternalSPSContext}}
used by *external* mode(standalone mode), which uses
{{DFSClient#listPaths(childPath,lastReturnedName, false);}} call for traversing
and collecting the file paths. (2){{IntraSPSNameNodeContext}} used by
*internal* mode, resides within Namenode and holds the reference of
{{namesystem}}, then traversing {{FSDirectory}} by making use of the existing
{{FSTreeTraverser.traverseDir}} functionality. So, internal sps has a
dependency with the namesystem, right?
With the above design, we have exposed a convenient private interface called
{{FileCollector}} and have provided only two specific implementations. The idea
of this private interface is just to make the code maintenance easy -
(1)ExternalSPSFilePathCollector for the *external* sps, which initializes and
holds the reference of {{DistributedFileSystem dfs}} (2)
IntraSPSNameNodeFileIdCollector for the *internal* sps, which holds the
reference of {{namesystem.getFSDirectory()}}. These implementations are
completely abstracted to specific Context and Context has respective API
exposed for the traversal functionality like below,
{code:java}
Context.java
/**
* Do scan and collects the files under that directory and adds to the given
* BlockStorageMovementNeeded.
*
* @param filePath
* file path
*/
void scanAndCollectFiles(long filePath)
throws IOException, InterruptedException;
{code}
{quote}More than one file collector implementation means the namesystem context
abstraction is being violated.
{quote}
Only two specific implementations, one should be for *external* context and
other one for strictly *internal* context. Like I said in the above comment, we
added {{FileCollector}} interface for better code readability. Lets imagine
that I haven't provided {{FileCollector}} interface, then the code looks like
{{ExternalSPSFilePathCollector}} implementation will be inlined into
ExternalSPSContext class and {{IntraSPSNameNodeFileIdCollector}} implementation
will be inlined into IntraSPSNameNodeContext class.
{noformat}
|-ExternalSPSContext#ExternalSPSFilePathCollector (Traverse
using DistributedFileSystem APIs)
|
Context-|
|
|-IntraSPSNameNodeContext#IntraSPSNameNodeFileIdCollector
(Traverse using NameSystem APIs)
{noformat}
{quote}SPS is not a true standalone service if there's two implementations of
various internal components like the file collector.
{quote}
Like I described above, file collection algo varies for *external* and
*internal*, in the way it fetches the details from Namenode. So, I believe the
exiting {{Context}} interface can abstract the specific implementation.
Welcome any better proposal/thoughts to unify both of these traversal logic
into single with minimal changes.
{quote}The overhead to maintain and test a tightly and loosely coupled version
will not be sustainable. The context shim must be the only pluggable piece.
Please provide a single implementation of the collector that leverages inode id
lookups via the context performing an inode path conversion.
{quote}
IIUC, your point is, why someone bothered about plugging in {{FileCollector}}.
Actually, we added {{FileCollector}} interface for better code readability and
only the Context interface is the pluggable piece and all the other specific
implementations are residing inside Context implementations. If
{{FileCollector}} interface is not conveying the purpose clearly, then I can
inline these code into the respective Context class, should I ?
With the [^HDFS-13381-HDFS-10285-01.patch] in this jira, you can see
{{IntraSPSNameNodeContext#fileCollector}} and
{{ExternalSPSContext#fileCollector}}. I hope you saw this patch.
{code:java}
***ExternalSPSContext.java***
public ExternalSPSContext(SPSService service, NameNodeConnector nnc) {
this.service = service;
this.nnc = nnc;
this.fileCollector = new ExternalSPSFilePathCollector(service);
this.externalHandler = new ExternalSPSBlockMoveTaskHandler(
service.getConf(), nnc, service);
this.blkMovementListener = new ExternalBlockMovementListener();
}
***IntraSPSNameNodeContext.java***
public IntraSPSNameNodeContext(Namesystem namesystem,
BlockManager blockManager, SPSService service) {
this.namesystem = namesystem;
this.blockManager =
[jira] [Commented] (HDFS-13245) RBF: State store DBMS implementation
[
https://issues.apache.org/jira/browse/HDFS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437425#comment-16437425
]
genericqa commented on HDFS-13245:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 33m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 40s{color} | {color:orange} hadoop-hdfs-project: The patch generated 21 new
+ 0 unchanged - 0 fixed = 21 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 57s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
38s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}227m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13245 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918931/HDFS-13245.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml findbugs checkstyle |
| uname | Linux 137e98825e3f 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|
[jira] [Commented] (HDFS-13330) ShortCircuitCache#fetchOrCreate never retries
[
https://issues.apache.org/jira/browse/HDFS-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437414#comment-16437414
]
Gabor Bota commented on HDFS-13330:
---
Thank you [~jojochuang]!
> ShortCircuitCache#fetchOrCreate never retries
> -
>
> Key: HDFS-13330
> URL: https://issues.apache.org/jira/browse/HDFS-13330
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HDFS-13330.001.patch, HDFS-13330.002.patch,
> HDFS-13330.004.patch, HDFS-13330.005.patch
>
>
> The follow do .. while(false) loop seems useless to me. The code intended to
> retry but it never worked. Let's fix it.
> {code:java:title=ShortCircuitCache#fetchOrCreate}
> ShortCircuitReplicaInfo info = null;
> do {
> if (closed) {
> LOG.trace("{}: can't fethchOrCreate {} because the cache is closed.",
> this, key);
> return null;
> }
> Waitable waitable = replicaInfoMap.get(key);
> if (waitable != null) {
> try {
> info = fetch(key, waitable);
> } catch (RetriableException e) {
> LOG.debug("{}: retrying {}", this, e.getMessage());
> }
> }
> } while (false);{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13433) webhdfs requests can be routed incorrectly in federated cluster
[
https://issues.apache.org/jira/browse/HDFS-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437379#comment-16437379
]
Arpit Agarwal commented on HDFS-13433:
--
v4 patch addresses checkstyle/whitespace issues. [~ajayydv] I also updated the
log message.
> webhdfs requests can be routed incorrectly in federated cluster
> ---
>
> Key: HDFS-13433
> URL: https://issues.apache.org/jira/browse/HDFS-13433
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Critical
> Attachments: HDFS-13433.01.patch, HDFS-13433.02.patch,
> HDFS-13433.03.patch, HDFS-13433.04.patch
>
>
> In the following HA+Federated setup with two nameservices ns1 and ns2:
> # ns1 -> namenodes nn1, nn2
> # ns2 -> namenodes nn3, nn4
> # fs.defaultFS is {{hdfs://ns1}}.
> A webhdfs request issued to nn3/nn4 will be routed to ns1. This is because
> {{setClientNamenodeAddress}} initializes {{NameNode#clientNamenodeAddress}}
> using fs.defaultFS before the config is overriden.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13433) webhdfs requests can be routed incorrectly in federated cluster
[
https://issues.apache.org/jira/browse/HDFS-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-13433:
-
Attachment: HDFS-13433.04.patch
> webhdfs requests can be routed incorrectly in federated cluster
> ---
>
> Key: HDFS-13433
> URL: https://issues.apache.org/jira/browse/HDFS-13433
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Critical
> Attachments: HDFS-13433.01.patch, HDFS-13433.02.patch,
> HDFS-13433.03.patch, HDFS-13433.04.patch
>
>
> In the following HA+Federated setup with two nameservices ns1 and ns2:
> # ns1 -> namenodes nn1, nn2
> # ns2 -> namenodes nn3, nn4
> # fs.defaultFS is {{hdfs://ns1}}.
> A webhdfs request issued to nn3/nn4 will be routed to ns1. This is because
> {{setClientNamenodeAddress}} initializes {{NameNode#clientNamenodeAddress}}
> using fs.defaultFS before the config is overriden.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13446) Ozone: Fix OzoneFileSystem contract test failures
[ https://issues.apache.org/jira/browse/HDFS-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-13446: - Status: Patch Available (was: Open) > Ozone: Fix OzoneFileSystem contract test failures > - > > Key: HDFS-13446 > URL: https://issues.apache.org/jira/browse/HDFS-13446 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-13446-HDFS-7240.001.patch > > > This jira refactors contract tests to the src/test directory and also fixes > the ozone filsystem contract tests as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13446) Ozone: Fix OzoneFileSystem contract test failures
Mukul Kumar Singh created HDFS-13446: Summary: Ozone: Fix OzoneFileSystem contract test failures Key: HDFS-13446 URL: https://issues.apache.org/jira/browse/HDFS-13446 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Mukul Kumar Singh Assignee: Mukul Kumar Singh Fix For: HDFS-7240 Attachments: HDFS-13446-HDFS-7240.001.patch This jira refactors contract tests to the src/test directory and also fixes the ozone filsystem contract tests as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13446) Ozone: Fix OzoneFileSystem contract test failures
[ https://issues.apache.org/jira/browse/HDFS-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-13446: - Attachment: HDFS-13446-HDFS-7240.001.patch > Ozone: Fix OzoneFileSystem contract test failures > - > > Key: HDFS-13446 > URL: https://issues.apache.org/jira/browse/HDFS-13446 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-13446-HDFS-7240.001.patch > > > This jira refactors contract tests to the src/test directory and also fixes > the ozone filsystem contract tests as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13310) [PROVIDED Phase 2] The DatanodeProtocol should be have DNA_BACKUP to backup blocks
[
https://issues.apache.org/jira/browse/HDFS-13310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437367#comment-16437367
]
Daryn Sharp commented on HDFS-13310:
{quote}The current BlockCommand protocol treats all blocks independently *since
DNs don't really have a concept of a file*; only blocks. This is the reason we
want a new command.
{quote}
The bolded (my emphasis) statement is accurately captures the crux of the
issue: is it a deficiency of the DN to only understand the concept of block?
The DN is currently has a simple and elegant design. It stores blocks. It
moves blocks. It deletes blocks. That's the design abstraction I implied will
become leaky.
That simplicity, which I believe is an excellent design strength, is at odds
with the design of this s4 upload feature. The DN must know the file id,
offset/length of the replica within the file, and block locations for an
unknown reason.
Here's my general concerns:
* Should the DN effectively become "file aware"? Perhaps it might be ok if
only for backup and only in the provided storage type.
* Will subsequent patches extend this file-awareness to more of the DN? If
yes, I have serious reservations.
* How will this functionality be managed? Do you intend to add the control
service directly into the NN?
* How will the feature interact with replication operations and the balancer?
Before debating the fine points, please help me understand the overall feature:
Is the intent that an admin must explicitly issue a "backup" operation? If
yes, what are the pros/cons over using a (modified) distcp?
> [PROVIDED Phase 2] The DatanodeProtocol should be have DNA_BACKUP to backup
> blocks
> --
>
> Key: HDFS-13310
> URL: https://issues.apache.org/jira/browse/HDFS-13310
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Major
> Attachments: HDFS-13310-HDFS-12090.001.patch,
> HDFS-13310-HDFS-12090.002.patch
>
>
> As part of HDFS-12090, Datanodes should be able to receive DatanodeCommands
> in the heartbeat response that instructs it to backup a block.
> This should take the form of two sub commands: PUT_FILE (when the file is <=1
> block in size) and MULTIPART_PUT_PART when part of a Multipart Upload (see
> HDFS-13186).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13445) Hostname is not getting resolved when we pass IP in webhdfs request.
[ https://issues.apache.org/jira/browse/HDFS-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13445: Component/s: security > Hostname is not getting resolved when we pass IP in webhdfs request. > > > Key: HDFS-13445 > URL: https://issues.apache.org/jira/browse/HDFS-13445 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, security, webhdfs >Affects Versions: 2.8.3 >Reporter: Ranith Sardar >Assignee: Ranith Sardar >Priority: Major > Attachments: HDFS-13445.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437338#comment-16437338
]
Daryn Sharp commented on HDFS-12703:
Unless someone else wants to provide a patch (note I didn't assign to myself
which means I don't intend to fix anytime soon), sure, move it out.
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Priority: Critical
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13422) Ozone: Fix whitespaces and license issues in HDFS-7240 branch
[
https://issues.apache.org/jira/browse/HDFS-13422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437272#comment-16437272
]
genericqa commented on HDFS-13422:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
54s{color} | {color:red} hadoop-ozone in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone hadoop-ozone/acceptance-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
56s{color} | {color:red} hadoop-hdds/common in HDFS-7240 has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
47s{color} | {color:red} hadoop-ozone/tools in HDFS-7240 has 5142 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
23s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
53s{color} | {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
52s{color} | {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone hadoop-ozone/acceptance-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
5s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} framework in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
27s{color} | {color:green} tools in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 58s{color}
| {color:red}
[jira] [Commented] (HDFS-13445) Hostname is not getting resolved when we pass IP in webhdfs request.
[ https://issues.apache.org/jira/browse/HDFS-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437271#comment-16437271 ] Ranith Sardar commented on HDFS-13445: -- I have attached my initial patch.[^HDFS-13445.001.patch] > Hostname is not getting resolved when we pass IP in webhdfs request. > > > Key: HDFS-13445 > URL: https://issues.apache.org/jira/browse/HDFS-13445 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, webhdfs >Affects Versions: 2.8.3 >Reporter: Ranith Sardar >Assignee: Ranith Sardar >Priority: Major > Attachments: HDFS-13445.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13445) Hostname is not getting resolved when we pass IP in webhdfs request.
[ https://issues.apache.org/jira/browse/HDFS-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ranith Sardar updated HDFS-13445: - Attachment: HDFS-13445.001.patch > Hostname is not getting resolved when we pass IP in webhdfs request. > > > Key: HDFS-13445 > URL: https://issues.apache.org/jira/browse/HDFS-13445 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, webhdfs >Affects Versions: 2.8.3 >Reporter: Ranith Sardar >Assignee: Ranith Sardar >Priority: Major > Attachments: HDFS-13445.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13281) Namenode#createFile should be /.reserved/raw/ aware.
[
https://issues.apache.org/jira/browse/HDFS-13281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437268#comment-16437268
]
Rushabh S Shah commented on HDFS-13281:
---
{quote}do we have plan to get this patch in shortly?
{quote}
I should have a patch by Monday or Tuesday. But I have changed the target
version to 2.8.5 since it is not blocker.
> Namenode#createFile should be /.reserved/raw/ aware.
>
>
> Key: HDFS-13281
> URL: https://issues.apache.org/jira/browse/HDFS-13281
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: encryption
>Affects Versions: 2.8.3
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Critical
> Attachments: HDFS-13281.001.patch, HDFS-13281.002.branch-2.patch,
> HDFS-13281.002.patch
>
>
> If I want to write to /.reserved/raw/ and if that directory happens to
> be in EZ, then namenode *should not* create edek and just copy the raw bytes
> from the source.
> Namenode#startFileInt should be /.reserved/raw/ aware.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13281) Namenode#createFile should be /.reserved/raw/ aware.
[ https://issues.apache.org/jira/browse/HDFS-13281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-13281: -- Target Version/s: 2.8.5 (was: 2.8.4) > Namenode#createFile should be /.reserved/raw/ aware. > > > Key: HDFS-13281 > URL: https://issues.apache.org/jira/browse/HDFS-13281 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: encryption >Affects Versions: 2.8.3 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Critical > Attachments: HDFS-13281.001.patch, HDFS-13281.002.branch-2.patch, > HDFS-13281.002.patch > > > If I want to write to /.reserved/raw/ and if that directory happens to > be in EZ, then namenode *should not* create edek and just copy the raw bytes > from the source. > Namenode#startFileInt should be /.reserved/raw/ aware. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13430) Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445
[
https://issues.apache.org/jira/browse/HDFS-13430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437264#comment-16437264
]
Rushabh S Shah commented on HDFS-13430:
---
{quote}I'll commit it to trunk on Friday noon PST if not.
{quote}
Sorry [~xiaochen] for not committing yet.
I am involved in an incident and haven't got time to go through the committer
docs.
So please go ahead and fix in trunk.
bq. You can still do the commit exercise with backports if you want.
Sounds good.
> Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445
> --
>
> Key: HDFS-13430
> URL: https://issues.apache.org/jira/browse/HDFS-13430
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HDFS-13430.01.patch
>
>
> Unfortunately HADOOP-14445 had an HDFS test failure that's not caught in the
> hadoop-common precommit runs.
> This is caught by our internal pre-commit using dist-test, and appears to be
> the only failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13445) Hostname is not getting resolved when we pass IP in webhdfs request.
[
https://issues.apache.org/jira/browse/HDFS-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437136#comment-16437136
]
Ranith Sardar edited comment on HDFS-13445 at 4/13/18 12:11 PM:
In the following snippet, it tries to get the server principal from the URL.If
we pass IP "KerberosAuthenticator.this.url.getHost()" will not resolved and
it's gives "HTTP/_HOST".
{code:java}
String servicePrincipal = KerberosUtil.getServicePrincipal("HTTP",
KerberosAuthenticator.this.url.getHost());
{code}
was (Author: ranith):
When we are trying to access the webhdfs with IP:port, it is not able to
resolve the HTTP/_HOST PRINCIPAL.
> Hostname is not getting resolved when we pass IP in webhdfs request.
>
>
> Key: HDFS-13445
> URL: https://issues.apache.org/jira/browse/HDFS-13445
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, webhdfs
>Affects Versions: 2.8.3
>Reporter: Ranith Sardar
>Assignee: Ranith Sardar
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13445) Hostname is not getting resolved when we pass IP in webhdfs request.
[ https://issues.apache.org/jira/browse/HDFS-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ranith Sardar updated HDFS-13445: - Summary: Hostname is not getting resolved when we pass IP in webhdfs request. (was: Web HDFS call is failing with spnego, when URL contain IP) > Hostname is not getting resolved when we pass IP in webhdfs request. > > > Key: HDFS-13445 > URL: https://issues.apache.org/jira/browse/HDFS-13445 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, webhdfs >Affects Versions: 2.8.3 >Reporter: Ranith Sardar >Assignee: Ranith Sardar >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12136) BlockSender performance regression due to volume scanner edge case
[
https://issues.apache.org/jira/browse/HDFS-12136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437193#comment-16437193
]
Wei-Chiu Chuang commented on HDFS-12136:
I believe this issue is no more after HDFS-11187 (fixed in 2.8.4)
> BlockSender performance regression due to volume scanner edge case
> --
>
> Key: HDFS-12136
> URL: https://issues.apache.org/jira/browse/HDFS-12136
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HDFS-12136.branch-2.patch, HDFS-12136.trunk.patch
>
>
> HDFS-11160 attempted to fix a volume scan race for a file appended mid-scan
> by reading the last checksum of finalized blocks within the {{BlockSender}}
> ctor. Unfortunately it's holding the exclusive dataset lock to open and read
> the metafile multiple times Block sender instantiation becomes serialized.
> Performance completely collapses under heavy disk i/o utilization or high
> xceiver activity. Ex. lost node replication, balancing, or decommissioning.
> The xceiver threads congest creating block senders and impair the heartbeat
> processing that is contending for the same lock. Combined with other lock
> contention issues, pipelines break and nodes sporadically go dead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13245) RBF: State store DBMS implementation
[ https://issues.apache.org/jira/browse/HDFS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiran Wu updated HDFS-13245: Status: Patch Available (was: Open) > RBF: State store DBMS implementation > > > Key: HDFS-13245 > URL: https://issues.apache.org/jira/browse/HDFS-13245 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: maobaolong >Assignee: Yiran Wu >Priority: Major > Attachments: HDFS-13245.001.patch, HDFS-13245.002.patch > > > Add a DBMS implementation for the State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13245) RBF: State store DBMS implementation
[ https://issues.apache.org/jira/browse/HDFS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiran Wu updated HDFS-13245: Attachment: HDFS-13245.002.patch > RBF: State store DBMS implementation > > > Key: HDFS-13245 > URL: https://issues.apache.org/jira/browse/HDFS-13245 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: maobaolong >Assignee: Yiran Wu >Priority: Major > Attachments: HDFS-13245.001.patch, HDFS-13245.002.patch > > > Add a DBMS implementation for the State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13245) RBF: State store DBMS implementation
[ https://issues.apache.org/jira/browse/HDFS-13245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiran Wu updated HDFS-13245: Status: Open (was: Patch Available) > RBF: State store DBMS implementation > > > Key: HDFS-13245 > URL: https://issues.apache.org/jira/browse/HDFS-13245 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: maobaolong >Assignee: Yiran Wu >Priority: Major > Attachments: HDFS-13245.001.patch, HDFS-13245.002.patch > > > Add a DBMS implementation for the State Store. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13444) Ozone: Fix checkstyle issues in HDFS-7240
[ https://issues.apache.org/jira/browse/HDFS-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437156#comment-16437156 ] genericqa commented on HDFS-13444: -- (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HDFS-Build/23923/console in case of problems. > Ozone: Fix checkstyle issues in HDFS-7240 > - > > Key: HDFS-13444 > URL: https://issues.apache.org/jira/browse/HDFS-13444 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-7240.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12747) Lease monitor may infinitely loop on the same lease
[
https://issues.apache.org/jira/browse/HDFS-12747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-12747:
--
Target Version/s: 2.8.5 (was: 2.8.4)
> Lease monitor may infinitely loop on the same lease
> ---
>
> Key: HDFS-12747
> URL: https://issues.apache.org/jira/browse/HDFS-12747
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
>
> Lease recovery incorrectly handles UC files if the last block is complete but
> the penultimate block is committed. Incorrectly handles is the euphemism for
> infinitely loops for days and leaves all abandoned streams open until
> customers complain.
> The problem may manifest when:
> # Block1 is committed but seemingly never completed
> # Block2 is allocated
> # Lease recovery is initiated for block2
> # Commit block synchronization invokes {{FSNamesytem#closeFileCommitBlocks}},
> causing:
> #* {{commitOrCompleteLastBlock}} to mark block2 as complete
> #*
> {{finalizeINodeFileUnderConstruction}}/{{INodeFile.assertAllBlocksComplete}}
> to throw {{IllegalStateException}} because the penultimate block1 is
> "COMMITTED but not COMPLETE"
> # The next lease recovery results in an infinite loop.
> The {{LeaseManager}} expects that {{FSNamesystem#internalReleaseLease}} will
> either init recovery and renew the lease, or remove the lease. In the
> described state it does neither. The switch case will break out if the last
> block is complete. (The case statement ironically contains an assert).
> Since nothing changed, the lease is still the “next” lease to be processed.
> The lease monitor loops for 25ms on the same lease, sleeps for 2s, loops on
> it again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12704) FBR may corrupt block state
[ https://issues.apache.org/jira/browse/HDFS-12704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437151#comment-16437151 ] Junping Du commented on HDFS-12704: --- move to 2.8.5 given issue has been quiet for a while. > FBR may corrupt block state > --- > > Key: HDFS-12704 > URL: https://issues.apache.org/jira/browse/HDFS-12704 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Priority: Critical > > If FBR processing generates a runtime exception it is believed to foul the > block state and lead to unpredictable behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12704) FBR may corrupt block state
[ https://issues.apache.org/jira/browse/HDFS-12704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-12704: -- Target Version/s: 2.8.5 (was: 2.8.4) > FBR may corrupt block state > --- > > Key: HDFS-12704 > URL: https://issues.apache.org/jira/browse/HDFS-12704 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Priority: Critical > > If FBR processing generates a runtime exception it is believed to foul the > block state and lead to unpredictable behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13445) Web HDFS call is failing with spnego, when URL contain IP
[ https://issues.apache.org/jira/browse/HDFS-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437136#comment-16437136 ] Ranith Sardar commented on HDFS-13445: -- When we are trying to access the webhdfs with IP:port, it is not able to resolve the HTTP/_HOST PRINCIPAL. > Web HDFS call is failing with spnego, when URL contain IP > - > > Key: HDFS-13445 > URL: https://issues.apache.org/jira/browse/HDFS-13445 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, webhdfs >Affects Versions: 2.8.3 >Reporter: Ranith Sardar >Assignee: Ranith Sardar >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7967) Reduce the performance impact of the balancer
[ https://issues.apache.org/jira/browse/HDFS-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437131#comment-16437131 ] Junping Du commented on HDFS-7967: -- [~kihwal] and [~daryn], do we need this patch on trunk/hadoop-3.x? If not, shall we make progress on branch-2.x only? > Reduce the performance impact of the balancer > - > > Key: HDFS-7967 > URL: https://issues.apache.org/jira/browse/HDFS-7967 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HDFS-7967-branch-2.8.patch, HDFS-7967-branch-2.patch, > HDFS-7967.branch-2-1.patch, HDFS-7967.branch-2.001.patch, > HDFS-7967.branch-2.002.patch, HDFS-7967.branch-2.8-1.patch, > HDFS-7967.branch-2.8.001.patch, HDFS-7967.branch-2.8.002.patch, > HDFS-7967.branch-2.8.003.patch > > > The balancer needs to query for blocks to move from overly full DNs. The > block lookup is extremely inefficient. An iterator of the node's blocks is > created from the iterators of its storages' blocks. A random number is > chosen corresponding to how many blocks will be skipped via the iterator. > Each skip requires costly scanning of triplets. > The current design also only considers node imbalances while ignoring > imbalances within the nodes's storages. A more efficient and intelligent > design may eliminate the costly skipping of blocks via round-robin selection > of blocks from the storages based on remaining capacity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13418) NetworkTopology should be configurable when enable DFSNetworkTopology
[
https://issues.apache.org/jira/browse/HDFS-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437129#comment-16437129
]
Hudson commented on HDFS-13418:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13992 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13992/])
HDFS-13418. NetworkTopology should be configurable when enable (yqlin: rev
0725953efec89b35b7586b846abb01f7c5963b37)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/net/DFSNetworkTopology.java
> NetworkTopology should be configurable when enable DFSNetworkTopology
> --
>
> Key: HDFS-13418
> URL: https://issues.apache.org/jira/browse/HDFS-13418
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.1
>Reporter: Tao Jie
>Assignee: Tao Jie
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 3.1.1
>
> Attachments: HDFS-13418-branch2.001.patch, HDFS-13418.001.patch,
> HDFS-13418.002.patch, HDFS-13418.003.patch, HDFS-13418.004.patch
>
>
> In HDFS-11530 we introduce DFSNetworkTopology and in HDFS-11998 we set
> DFSNetworkTopology as the default implementation.
> We still have {{net.topology.impl=org.apache.hadoop.net.NetworkTopology}} in
> core-site.default. Actually this property does not effect once
> {{dfs.use.dfs.network.topology}} is true.
> in {{DatanodeManager}}，networkTopology is initialized as
> {code}
> if (useDfsNetworkTopology) {
> networktopology = DFSNetworkTopology.getInstance(conf);
> } else {
> networktopology = NetworkTopology.getInstance(conf);
> }
> {code}
> I think we should still make the NetworkTopology configurable rather than
> hard code the implementation since we may need another NetworkTopology impl.
> I am not sure if there is other consideration. Any thought? [~vagarychen]
> [~linyiqun]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13281) Namenode#createFile should be /.reserved/raw/ aware.
[ https://issues.apache.org/jira/browse/HDFS-13281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437122#comment-16437122 ] Junping Du commented on HDFS-13281: --- Hi [~shahrs87] and [~xiaochen], do we have plan to get this patch in shortly? If not, I will move it to 2.8.5 to unblock 2.8.4 release. > Namenode#createFile should be /.reserved/raw/ aware. > > > Key: HDFS-13281 > URL: https://issues.apache.org/jira/browse/HDFS-13281 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: encryption >Affects Versions: 2.8.3 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Critical > Attachments: HDFS-13281.001.patch, HDFS-13281.002.branch-2.patch, > HDFS-13281.002.patch > > > If I want to write to /.reserved/raw/ and if that directory happens to > be in EZ, then namenode *should not* create edek and just copy the raw bytes > from the source. > Namenode#startFileInt should be /.reserved/raw/ aware. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
