[jira] [Commented] (HDFS-10314) A new tool to sync current HDFS view to specified snapshot

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530595#comment-15530595
 ] 

Yongjun Zhang commented on HDFS-10314:
--

Had a discussion with [~jingzhao], and we had the following agreement:

1. For now, he will be fine with option 2 stated in

https://issues.apache.org/jira/browse/HDFS-10314?focusedCommentId=15524359=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15524359

as long as we document it well, even though it's not his favorite. In that 
case, we can continue to work on HDFS-9820. 

2. When creating a new tool in the future (HDFS-10314), we need to do the 
following: 
* refactor the DistCp code to separate out the snapshot sync part (that handles 
rename/delete per snapshot diff) and copyList calculation part to its own 
class, e.g., DistCpPrepare. 
* let both DistCp and DistSync to call DistCpPrepare for the functionality they 
need
* Modify DistCp to take an optional new argument copyListing.
* Let DistSync call DistCpPrepare to do the snapshot sync part and copyListing 
creation part, and then pass the copyListing to DIstCp.

Please feel free to correct/add if I'm inaccurate or missed anything.

Thanks much Jing.


> A new tool to sync current HDFS view to specified snapshot
> --
>
> Key: HDFS-10314
> URL: https://issues.apache.org/jira/browse/HDFS-10314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-10314.001.patch
>
>
> HDFS-9820 proposed adding -rdiff switch to distcp, as a reversed operation of 
> -diff switch. 
> Upon discussion with [~jingzhao], we will introduce a new tool that wraps 
> around distcp to achieve the same purpose.
> I'm thinking about calling the new tool "rsync", similar to unix/linux 
> command "rsync". The "r" here means remote.
> The syntax that simulate -rdiff behavior proposed in HDFS-9820 is
> {code}
> rsync  
> {code}
> This command ensure   is newer than .
> I think, In the future, we can add another command to have the functionality 
> of -diff switch of distcp.
> {code}
> sync  
> {code}
> that ensures   is older than .
> Thanks [~jingzhao].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10314) A new tool to sync current HDFS view to specified snapshot

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528169#comment-15528169
 ] 

Yongjun Zhang commented on HDFS-10314:
--

HI [~jingzhao],

I tried to provide details in my earlier replies so they tend to be lengthy. 
Now I'd like to ask couple of quick questions on your latest proprosal (I asked 
too in earlier reply but too buried). Appreciate your taking time to reply.

{quote}
In that sense, I think a simpler way is to wrap (but not extend) the current 
distcp in the snapshot-restore tool:
1. The tool takes a single cluster and a target snapshot as arguments
2. The tool computes the delta for restoring using snapshot diff report
3. The tool does rename/delete etc. metadata ops to revert part of the diff
4. The tool uses the distcp (by invokes distcp as a library) to copy the 
original states of modified files
{quote}

Q1: step 2 does the snapshot diff calculation as you described, does it also 
collect the modified files and pass to step 4?

Q2: or does step 4 also do snapshot calculation? 
 
Thanks much.



> A new tool to sync current HDFS view to specified snapshot
> --
>
> Key: HDFS-10314
> URL: https://issues.apache.org/jira/browse/HDFS-10314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-10314.001.patch
>
>
> HDFS-9820 proposed adding -rdiff switch to distcp, as a reversed operation of 
> -diff switch. 
> Upon discussion with [~jingzhao], we will introduce a new tool that wraps 
> around distcp to achieve the same purpose.
> I'm thinking about calling the new tool "rsync", similar to unix/linux 
> command "rsync". The "r" here means remote.
> The syntax that simulate -rdiff behavior proposed in HDFS-9820 is
> {code}
> rsync  
> {code}
> This command ensure   is newer than .
> I think, In the future, we can add another command to have the functionality 
> of -diff switch of distcp.
> {code}
> sync  
> {code}
> that ensures   is older than .
> Thanks [~jingzhao].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10314) A new tool to sync current HDFS view to specified snapshot

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524359#comment-15524359
 ] 

Yongjun Zhang commented on HDFS-10314:
--

Hi [~jingzhao],

To better address your comments, I discussed with [~atm], and we think there 
are the following possible options:

Option 1. To get closer to what you suggested, we can introduce new tool 
distsync and let it support all of the following semantics:
* distsync -diff s1 s2 src tgt
* distsync -rdiff s2 s1 src tgt
* distsync tgt s1

Implementation-wise, we can move common DistCp stuff to AbstractDistCp, and let 
both DistCp and DistSync inherit from AbstractDistCp.  Accordingly, 
DistCpOption need to be refactored in a similar fashion.

With this route, the "-diff" in distcp will probably eventually be obsoleted.

Option 2. Given that we already have -diff switch in DistCp, we can add 
"-rdiff" to DistCp, and support
* distcp -diff s1 s2 src tgt (currently supported)
* distcp -rdiff s2 s1 src tgt

Would you please share what you think?

Thanks a lot.








> A new tool to sync current HDFS view to specified snapshot
> --
>
> Key: HDFS-10314
> URL: https://issues.apache.org/jira/browse/HDFS-10314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-10314.001.patch
>
>
> HDFS-9820 proposed adding -rdiff switch to distcp, as a reversed operation of 
> -diff switch. 
> Upon discussion with [~jingzhao], we will introduce a new tool that wraps 
> around distcp to achieve the same purpose.
> I'm thinking about calling the new tool "rsync", similar to unix/linux 
> command "rsync". The "r" here means remote.
> The syntax that simulate -rdiff behavior proposed in HDFS-9820 is
> {code}
> rsync  
> {code}
> This command ensure   is newer than .
> I think, In the future, we can add another command to have the functionality 
> of -diff switch of distcp.
> {code}
> sync  
> {code}
> that ensures   is older than .
> Thanks [~jingzhao].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10314) A new tool to sync current HDFS view to specified snapshot

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512098#comment-15512098
 ] 

Yongjun Zhang commented on HDFS-10314:
--

Hi [~jingzhao],

Thanks again for your earlier feedback. 

Would like to share the details below about why I don't think your proposed 
method is simpler. Hope it makes some sense to you, and please correct me if 
I'm wrong. I hope you could elaborate here to help me understand better.

DistCp does two basic steps:
# based on the input, create the copyListing, which is a sequence file for 
mapreduce, and each entry contains info to  figure out one pair of  and file attribute info
# throw the sequence file to the mapreduce job

Step 2 is relatively stable these days, mostly we are manipulating step 1 based 
on the input. 

"-diff s1 s2" replaced the original step 1 with a new step 1:
* 1.1 compute snapshot diff, 
* 1.2 figure out the rename/delete operation's source and target, based on the 
snapshot diff info
* 1.3 apply the rename/delete to the target path
* 1.4 figure out the add/modification operation's source and target, based on 
the snapshot diff info
* 1.5 create copyListing based on step 1.4

*The tricky parts* are 1,2 and 1.4, and the order of applying the rename/delete 
operations in step 1.3.  With HDFS-7535 and HDFS-8828, a framework has been 
implemented in DistCp that does the new step 1. What I did was to re-use the 
framework.

Now the questions:

# With what you proposed, I don't see how the tricky parts I listed above are 
simplified. And you suggested not to touch existing DistCp implementation, I 
thought you meant to rewrite the code that does the tricky parts, which is not 
simpler.  
# Which step in your proposal will generate the copyListing? Step 3 or step 4? 
** If it's in step 3, how we are going to pass the result to distcp in step 4?
** or if it's in step 4, that means we need to calculate the snapshot diff 
again in step 4, and do the tricky manipulation again there. It doesn't look 
simpler, and probably additional access to NN.
# Would you please share the specific problems you see with my implementation, 
other than you think your proposal would be simpler? I really hope you could do 
that.

Thanks much.
















> A new tool to sync current HDFS view to specified snapshot
> --
>
> Key: HDFS-10314
> URL: https://issues.apache.org/jira/browse/HDFS-10314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-10314.001.patch
>
>
> HDFS-9820 proposed adding -rdiff switch to distcp, as a reversed operation of 
> -diff switch. 
> Upon discussion with [~jingzhao], we will introduce a new tool that wraps 
> around distcp to achieve the same purpose.
> I'm thinking about calling the new tool "rsync", similar to unix/linux 
> command "rsync". The "r" here means remote.
> The syntax that simulate -rdiff behavior proposed in HDFS-9820 is
> {code}
> rsync  
> {code}
> This command ensure   is newer than .
> I think, In the future, we can add another command to have the functionality 
> of -diff switch of distcp.
> {code}
> sync  
> {code}
> that ensures   is older than .
> Thanks [~jingzhao].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10314) A new tool to sync current HDFS view to specified snapshot

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510391#comment-15510391
 ] 

Yongjun Zhang commented on HDFS-10314:
--

Hi [~jingzhao],

For clarity, and as a recap, here is a comparison table between -diff and the 
proposed -rdiff, which shows the symmetricity:

||Comparison||-diff s1 s2  ||-rdiff s2 s1  ||
|Current feature state|Existing in distcp|Proposed Addition |
|Functionality| Given 's current state is s1, make 's current state 
the same as newer snapshot s2 | Given 's current state is s2, make 's 
current state the same as older snapshot s1 | 
|Requirements| #  and  need to be different paths
# both  and  have snapshot s1 with exact same content 
#  has snapshot s2
# s2 is newer than s1
# 's current state is the same as s1
#  doesn't have snapshot s2 | #  and  can be the same or 
different paths
# both  and  have snapshot s1 with exact same content
#  has snapshot s2
#  s2 is newer than s1 
# 's current state is the same as s2
#  may or may not have snapshot s2 |
|Steps|# calculate snapshotDiff at  
# apply rename/delete part of snapshotDiff on  
# copy modified part of snapshotDiff from s1 of  to  | # calculate 
snapshotDiff at  
# apply rename/delete part of snapshotDiff on  
# copy modified part of snapshotDiff from s1 of  to  |

The original thinking was to add -ridff to distcp (solution A), but because of 
the concern of confusing semantics, it's suggested to introduce a new command 
here (solution B). 

Thanks.


> A new tool to sync current HDFS view to specified snapshot
> --
>
> Key: HDFS-10314
> URL: https://issues.apache.org/jira/browse/HDFS-10314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-10314.001.patch
>
>
> HDFS-9820 proposed adding -rdiff switch to distcp, as a reversed operation of 
> -diff switch. 
> Upon discussion with [~jingzhao], we will introduce a new tool that wraps 
> around distcp to achieve the same purpose.
> I'm thinking about calling the new tool "rsync", similar to unix/linux 
> command "rsync". The "r" here means remote.
> The syntax that simulate -rdiff behavior proposed in HDFS-9820 is
> {code}
> rsync  
> {code}
> This command ensure   is newer than .
> I think, In the future, we can add another command to have the functionality 
> of -diff switch of distcp.
> {code}
> sync  
> {code}
> that ensures   is older than .
> Thanks [~jingzhao].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10314) A new tool to sync current HDFS view to specified snapshot

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508167#comment-15508167
 ] 

Yongjun Zhang commented on HDFS-10314:
--

Many thanks [~jingzhao] for the review and feedback. Please see my answers 
below.

{quote}
So the current patch actually adds a new distsync extension and implements the 
"calculating diff on target cluster" approach? 
{quote}
Yes. This is the result of our discussion in HDFS-9820. 

Though I preferred adding -rdiff as a symmetric behavior as how -diff works in 
distcp, as reported in HDFS-9820, I think your suggestion of creating a new 
tool is fine, as long as we leverage the code that does -diff in distcp, and 
minimize code duplication.

{quote}
I think to have the diff calculated on target is fine, 
{quote}
Yes. Since the goal is to make the target's state go to a specified snapshot,  
we'd better calculate snapshot diff at the target.

{quote}
but I'm not sure to directly extend the current distcp is a good idea.
{quote}
There are couple of reasons when I came up with the idea of extending distcp:
* distsync is a customized distcp,  it extends distcp's -diff behavior to 
support -rdiff.
* it's better to re-use the code that implements -diff, extending allows 
re-using the existing implementation of "-diff". You can see it's only 124 
lines of code (including the header and imports) in DistSync.java in my patch 
rev001.
 
{quote}
Correct me if I'm wrong. Here's my current understanding of the patch:
1. our main motivation is still to utilize distcp to restore a snapshot
2. the idea is to compute the delta on the target cluster, and for modified 
files we get their original state from the source.
{quote}
Yes. However, for modified files, I intended to make it flexible to copy from 
the specified snapshot of either the source or the target.

{quote} 
In that sense, I think a simpler way is to wrap (but not extend) the current 
distcp in the snapshot-restore tool:
1. The tool takes a single cluster and a target snapshot as arguments
2. The tool computes the delta for restoring using snapshot diff report
3. The tool does rename/delete etc. metadata ops to revert part of the diff
4. The tool uses the distcp (by invokes distcp as a library) to copy the 
original states of modified files
In this way we can minimize the change (no need to touch the current distcp 
implementation/arguments), and provides a new tool with simple but clear 
semantic. We may lose some flexibility (only handling one cluster) but the tool 
itself will be easy to use and will not cause any confusion to the end users.
What do you think? Please let me know if I miss anything.
{quote}

We discussed two overall solutions earlier.

* Solution A. What proposed in HDFS-9820: adding "-rdiff s2 s1" to distcp, to 
achieve the symmetric behavior as "-diff s1 s2" of distcp.
* Solution B. What proposed in HDFS-10314: introducing a new tool, that allows 
to sync a target cluster to a specified snapshot.

For Solution B,  there are two approaches, one (B.1) is my patch rev001 here, 
the other (B.2) is what you proposed above. 

Some thoughts:

# Creating a new tool itself is going to mean extra support, that's why I 
preferred solution #A, which is the simplest.
# Given that we want to create a new tool, we'd better maximize code sharing, 
otherwise, it's going to be both more development effort and extra support 
effort. 
# To me, the way suggested by solution #B.2 disallows sharing the existing 
implementation of -diff in distcp. Thus I think it's actually not simpler, and 
would incur support burden for future because of the duplicated code.
# I think we agreed per our discussion that if we create a new tool, then you 
don't have strong opinion whether we copy from a different cluster or from the 
same target cluster. As I shared earlier, I can tell from the user's case, that 
copying from a different mirror cluster can be much faster sometimes. So I kept 
suggesting that it would be better to support the flexibility, to copy from 
either the source or the target.

Would you please kindly share the specific problems you see with solution #B.1? 

Honestly speaking, I still prefer solution #A. But I'm ok with solution B, 
except I really hope to share the code of -diff implemented in distcp already.

Thanks a lot.



> A new tool to sync current HDFS view to specified snapshot
> --
>
> Key: HDFS-10314
> URL: https://issues.apache.org/jira/browse/HDFS-10314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-10314.001.patch
>
>
> HDFS-9820 proposed adding -rdiff switch to distcp, as a reversed operation of 
> -diff switch. 
> Upon discussion with [~jingzhao], we will introduce a new tool 

[jira] [Commented] (HDFS-10314) A new tool to sync current HDFS view to specified snapshot

[Jing Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507688#comment-15507688
 ] 

Jing Zhao commented on HDFS-10314:
--

So the current patch actually adds a new distsync extension and implements the 
"calculating diff on target cluster" approach? I think to have the diff 
calculated on target is fine, but I'm not sure to directly extend the current 
distcp is a good idea.

Correct me if I'm wrong. Here's my current understanding of the patch:
1. our main motivation is still to utilize distcp to restore a snapshot
2. the idea is to compute the delta on the target cluster, and for modified 
files we get their original state from the source. 

In that sense, I think a simpler way is to wrap (but not extend) the current 
distcp in the snapshot-restore tool:
1. The tool takes a single cluster and a target snapshot as arguments
2. The tool computes the delta for restoring using snapshot diff report
3. The tool does rename/delete etc. metadata ops to revert part of the diff
4. The tool uses the distcp (by invokes distcp as a library) to copy the 
original states of modified files

In this way we can minimize the change (no need to touch the current distcp 
implementation/arguments), and provides a new tool with simple but clear 
semantic. We may lose some flexibility (only handling one cluster) but the tool 
itself will be easy to use and will not cause any confusion to the end users.

What do you think? Please let me know if I miss anything.


> A new tool to sync current HDFS view to specified snapshot
> --
>
> Key: HDFS-10314
> URL: https://issues.apache.org/jira/browse/HDFS-10314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-10314.001.patch
>
>
> HDFS-9820 proposed adding -rdiff switch to distcp, as a reversed operation of 
> -diff switch. 
> Upon discussion with [~jingzhao], we will introduce a new tool that wraps 
> around distcp to achieve the same purpose.
> I'm thinking about calling the new tool "rsync", similar to unix/linux 
> command "rsync". The "r" here means remote.
> The syntax that simulate -rdiff behavior proposed in HDFS-9820 is
> {code}
> rsync  
> {code}
> This command ensure   is newer than .
> I think, In the future, we can add another command to have the functionality 
> of -diff switch of distcp.
> {code}
> sync  
> {code}
> that ensures   is older than .
> Thanks [~jingzhao].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10314) A new tool to sync current HDFS view to specified snapshot

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505086#comment-15505086
 ] 

Yongjun Zhang commented on HDFS-10314:
--

BTW [~jingzhao],

The usage of the solution is:

1. distsync -diff s1 s2 src tgt

This will be exactly the same behavior as "distcp -diff s1 s2 src tgt", it will 
copy the delta changes made in src (between snapshot s1 and s2) to target.

It requires that snapshot s1 to be older than s2

2. distsync -rdiff s2 s1 src tgt

snapshot s2 has to be newer than s1; 

That is, since the creation of s1 on tgt, some changes are made on tgt, and 
these changes are captured in s2.

The goal of this command is to make tgt go back to s1, by reverting the changes 
between s1 and s2, and copy the necessary files from s1 of src.

The src could be the same as tgt, or different as we discussed earlier. When 
the src is different than tgt, src's s1 is expected to be the exact content as 
s1 of tgt. So we can copy files from either s1 of src, or s1 of tgt.

Would you please take a look?

Thanks a lot!


> A new tool to sync current HDFS view to specified snapshot
> --
>
> Key: HDFS-10314
> URL: https://issues.apache.org/jira/browse/HDFS-10314
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-10314.001.patch
>
>
> HDFS-9820 proposed adding -rdiff switch to distcp, as a reversed operation of 
> -diff switch. 
> Upon discussion with [~jingzhao], we will introduce a new tool that wraps 
> around distcp to achieve the same purpose.
> I'm thinking about calling the new tool "rsync", similar to unix/linux 
> command "rsync". The "r" here means remote.
> The syntax that simulate -rdiff behavior proposed in HDFS-9820 is
> {code}
> rsync  
> {code}
> This command ensure   is newer than .
> I think, In the future, we can add another command to have the functionality 
> of -diff switch of distcp.
> {code}
> sync  
> {code}
> that ensures   is older than .
> Thanks [~jingzhao].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10314) A new tool to sync current HDFS view to specified snapshot

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498586#comment-15498586
 ] 

Hadoop QA commented on HDFS-10314:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
26s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue}  0m  
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  9m 
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 44s{color} | {color:orange} root: The patch generated 61 new + 175 unchanged 
- 9 fixed = 236 total (was 184) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m 
14s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
17s{color} | {color:red} hadoop-tools_hadoop-distcp generated 2 new + 50 
unchanged - 0 fixed = 52 total (was 50) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m 
50s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 37s{color} 
| {color:red} hadoop-distcp in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m  0s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.TestOptionsParser |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10314 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12828993/HDFS-10314.001.patch |
| Optional Tests |  asflicense  mvnsite  unit  compile  javac  javadoc  
mvninstall  findbugs  checkstyle  shellcheck  shelldocs  |
| uname | Linux daea70b13ae9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 4174b97 |
| Default Java | 1.8.0_101 |
| shellcheck | v0.4.4 |
| findbugs | v3.0.0 |
| checkstyle |