[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904501#comment-16904501
]
Hadoop QA commented on HADOOP-16112:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 13 new + 165 unchanged - 0 fixed = 178 total (was 165) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
47s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
10s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | Spinning on TrashPolicyDefault.flag in
org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(Path) At
TrashPolicyDefault.java: At TrashPolicyDefault.java:[line 168] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HADOOP-16112 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977234/HADOOP-16112.004.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5fa4cd19483b 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d29007f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16466/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| findbugs |
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904489#comment-16904489
]
Lisheng Sun commented on HADOOP-16112:
--
upload v4 patch.
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch,
> HADOOP-16112.003.patch, HADOOP-16112.004.patch, HADOOP-16504.000.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900565#comment-16900565
]
Lisheng Sun commented on HADOOP-16112:
--
hi [~jojochuang] Could you mind review this patch? Thank you.
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch,
> HADOOP-16112.003.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899795#comment-16899795
]
Lisheng Sun commented on HADOOP-16112:
--
hi [Wei-Chiu
Chuang|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=jojochuang]
[~elgoiri] [~aajisaka] [~ayushtkn] Could you have time to review this patch?
Thank you a lot.
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch,
> HADOOP-16112.003.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899603#comment-16899603
]
Hadoop QA commented on HADOOP-16112:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 10 new + 157 unchanged - 0 fixed = 167 total (was 157) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
8s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HADOOP-16112 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12976633/HADOOP-16112.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 29333c912cc2 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 065cbc6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16457/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16457/testReport/ |
| Max. process+thread count | 1361 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16457/console |
| Powered by | Apache
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863704#comment-16863704
]
Lisheng Sun commented on HADOOP-16112:
--
[~jzhuge] [~hexiaoqiao] , Thanks for your comments.
My users hit this issue more once.If the unexpected trash location generated,
lead to exist directory is deleted. In fact, this situation don't add timestamp
to mkdir. This result should be not expected.
If there is a issue, I will fix unit test in which race condition reproduce and
add a patch. Please correct me if I am wrong.Thanks.
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863689#comment-16863689
]
He Xiaoqiao commented on HADOOP-16112:
--
[~jzhuge], Thanks for your comments.
{quote}the new unit test passes without any fix, is it valid? I understand race
condition is hard to reproduce.{quote}
right, new unit test actually does not verify anything, so in my opinion, it is
not a valid unit test.
I would like to state this case could be interpretable, may be not issue. IMO
it make sense for me no matter parent or child path add timestamp to mkdir, in
whatever way, I do not think we could guarantee consistency at the client side
through retry. Please correct me if something wrong.
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863683#comment-16863683
]
John Zhuge commented on HADOOP-16112:
-
[~leosun08], thanks for finding and reporting the issue! I can take a look.
Just echo [~hexiaoqiao]'s comment, the new unit test passes without any fix, is
it valid? I understand race condition is hard to reproduce.
BTW, have you or your users hit this issue often? What was the impact other
than being surprised by the unexpected trash location?
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862811#comment-16862811
]
Lisheng Sun commented on HADOOP-16112:
--
[~hexiaoqiao] Thanks for review. I think it could be issue.
Hi [~ferhui] [~jzhuge] Mind you help to review this issue, Thanks.
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862802#comment-16862802
]
He Xiaoqiao commented on HADOOP-16112:
--
[~leosun08] thanks for your report and sorry for late comment. just quick
review #moveToTrash logic, I think the case you mentioned could be
interpretable, may be not issue. [~ferhui] [~jzhuge] should be more proper to
give another reviews.
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861705#comment-16861705
]
Lisheng Sun commented on HADOOP-16112:
--
[~hexiaoqiao] The current result should be not expected result. Please correct
me if I am wrong. I am confused about this issue. Thanks.
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860469#comment-16860469
]
Lisheng Sun commented on HADOOP-16112:
--
[~hexiaoqiao]
{code:java}
I mean the trash directory should be
{/user/test/.Trash/Current/user/test/a+timestamp/b}.
But when the code catch FileAlreadyExistsException, another thread delete
/user/test/.Trash/Current/user/test/a in this corner case before the code
Path existsFilePath = baseTrashPath;
while (!fs.exists(existsFilePath)) {
existsFilePath = existsFilePath.getParent();
}
So the actual result is /user/test/.Trash/Current/user/test+timestamp/a/b. It
is not expected result.{code}
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860040#comment-16860040
]
He Xiaoqiao commented on HADOOP-16112:
--
[~leosun08], Do you mean that trash directory should be
{/user/test/.Trash/Current/user/test/a+timestamp/b} rather than
{/user/test/.Trash/Current/user/test+timestamp/a/b} which is actual result in
some corner case?
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859949#comment-16859949
]
Lisheng Sun commented on HADOOP-16112:
--
[~hexiaoqiao] , thanks for the reply.
{code:java}
fs -mkdir /user/test
fs -mkdir /user/test/a
fs -mkdir /user/test/a/b fail
catch
fs -rm -r/user/test/a (another thread)
fs -mkdir /user/test/a/b
{code}
The current code is intented to mkdir /user/test/a+timestamp/b, but in actual
mkdir /user/test+timestamp/a/b.
The race condition is reproduced hard by Unit Test.
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859920#comment-16859920
]
He Xiaoqiao commented on HADOOP-16112:
--
Sorry I don't catch the point and just verify the unit test you attach in
[^HADOOP-16112.002.patch], and it run correct, So I am confused what this issue
means?
Would you mind offer how to reproduce this issue? for instance(just example, no
more info.)
{code:java}
fs -mkdir /user/test
fs -rm -r /user/test
fs -mkdir /user/test/a
fs -rm -r /user/test/b (another thread)
{code}
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859912#comment-16859912
]
Hadoop QA commented on HADOOP-16112:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 14s{color} | {color:orange} root: The patch generated 4 new + 9 unchanged -
0 fixed = 13 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
27s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 1s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
55s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}222m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestMultipleNNPortQOP |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication |
| | hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile
|
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HADOOP-16112 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959369/HADOOP-16112.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e3caaa79c523 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar
18 15:02:05 UTC 2019 x86_64 x86_64 x86_64
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859794#comment-16859794
]
Lisheng Sun commented on HADOOP-16112:
--
Thanks for watching this issue.
The issue is a race condition in TrashPolicyDefault#moveToTrash.
try {
if (!fs.mkdirs(baseTrashPath, PERMISSION)) { // create current
LOG.warn("Can't create(mkdir) trash directory: " + baseTrashPath);
return false;
}
} catch (FileAlreadyExistsException e) {
// find the path which is not a directory, and modify baseTrashPath
// & trashPath, then mkdirs
Path existsFilePath = baseTrashPath;
while (!fs.exists(existsFilePath)) {
existsFilePath = existsFilePath.getParent();
}
baseTrashPath = new Path(baseTrashPath.toString().replace(
existsFilePath.toString(), existsFilePath.toString() + Time.now())
);
trashPath = new Path(baseTrashPath, trashPath.getName());
// retry, ignore current failure
--i;
continue;
} catch (IOException e) {
LOG.warn("Can't create trash directory: " + baseTrashPath, e);
cause = e;
break;
}
try {
// if the target path in Trash already exists, then append with
// a current time in millisecs.
String orig = trashPath.toString();
while(fs.exists(trashPath)) {
trashPath = new Path(orig + Time.now());
}
// move to current trash
fs.rename(path, trashPath,
Rename.TO_TRASH);
LOG.info("Moved: '" + path + "' to trash at: " + trashPath);
return true;
} catch (IOException e) {
cause = e;
}
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859749#comment-16859749
]
He Xiaoqiao commented on HADOOP-16112:
--
[~leosun08] Thanks for your reports. Would you mind describe this issue
detailed and how to reproduce?
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859739#comment-16859739
]
Lisheng Sun commented on HADOOP-16112:
--
[~jzhuge] [~ste...@apache.org] Could you please take a look?
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839141#comment-16839141
]
Lisheng Sun commented on HADOOP-16112:
--
Test failures are not caused by this patch.
> Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
> -
>
> Key: HADOOP-16112
> URL: https://issues.apache.org/jira/browse/HADOOP-16112
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.2.0
>Reporter: Lisheng Sun
>Priority: Major
> Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch
>
>
> There is race condition in TrashPolicyDefault#moveToTrash
> try {
> if (!fs.mkdirs(baseTrashPath, PERMISSION))
> { // create current LOG.warn("Can't create(mkdir) trash directory: " +
> baseTrashPath); return false; }
> } catch (FileAlreadyExistsException e) {
> // find the path which is not a directory, and modify baseTrashPath
> // & trashPath, then mkdirs
> Path existsFilePath = baseTrashPath;
> while (!fs.exists(existsFilePath))
> { existsFilePath = existsFilePath.getParent(); }
> {color:#ff}// case{color}
> {color:#ff} other thread deletes existsFilePath here ，the results
> doesn't meet expectation{color}
> {color:#ff} for example{color}
> {color:#ff} there is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color}
> {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is
> deleted, the result is
> /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color}
> {color:#ff} so when existsFilePath is deleted, don't modify
> baseTrashPath. {color}
> baseTrashPath = new Path(baseTrashPath.toString().replace(
> existsFilePath.toString(), existsFilePath.toString() + Time.now())
> );
> trashPath = new Path(baseTrashPath, trashPath.getName());
> // retry, ignore current failure
> --i;
> continue;
> } catch (IOException e)
> { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e;
> break; }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[
https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772831#comment-16772831
]
Hadoop QA commented on HADOOP-16112:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 28s{color} | {color:orange} root: The patch generated 4 new + 9 unchanged -
0 fixed = 13 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
3s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 57s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}219m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16112 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959369/HADOOP-16112.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f417afbc7622 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1d30fd9 |
| maven | version: Apache Maven 3.3.9 |
| Default
