[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=740177=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-740177
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 11/Mar/22 17:40
Start Date: 11/Mar/22 17:40
Worklog Time Spent: 10m
Work Description: steveloughran edited a comment on pull request #2735:
URL: https://github.com/apache/hadoop/pull/2735#issuecomment-892760289
I have stopped working on this. Feel free to take it up
I originally thought "hey, we could just make this public and there'd be a
good rename", but as usual the challenge becomes one of strictly implementing
the preconditions. FileContext does that, though non-atomically; Factoring out
all that policy at least makes things consistent.
But, having dealt with other rename related trouble recently, I'm thinking
really I'd want a new builder-based rename
```
Future foutcome = FS.renamePath(source, dest)
.must("fs.opt.rename.atomic", true)
.may("fs.opt.rename.etag", "afee338b")
.build()
RenameOutcome outcome = outcome.get();
class RenameOutcome implements IOStatisticsSource {
}
Why this?
* Allows for stores which count IO Costs of renames to report them
* You need some kind of return type for java futures
* builder options would let you say whether you MUST have atomic rename, in
which case
-no s3a, wasb or gcs rename for you.
* add ability to pass in things like etags
Why async?
* so slow stores can be obviously slow about it
* let you pass in a progressable. Distcp could do this so stop tasks failing
during rename of big files on non -direct uploads to s3; same for
FileOutputCommitter
```
Future foutcome = FS.renamePath(source, dest)
.withProgress(reporter)
.build()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 740177)
Time Spent: 4h 40m (was: 4.5h)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=671493=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671493
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 28/Oct/21 15:06
Start Date: 28/Oct/21 15:06
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #2735:
URL: https://github.com/apache/hadoop/pull/2735#issuecomment-826510805
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 21m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 9 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 50s | | trunk passed |
| +1 :green_heart: | compile | 22m 15s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 18m 59s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 8s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 20s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 9m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 30s | | the patch passed |
| +1 :green_heart: | compile | 21m 35s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 35s | |
root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 0 new + 1926 unchanged - 5
fixed = 1926 total (was 1931) |
| +1 :green_heart: | compile | 19m 1s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 19m 1s | |
root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 0 new + 1825 unchanged -
5 fixed = 1825 total (was 1830) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/4/artifact/out/blanks-eol.txt)
| The patch has 7 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 59s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/4/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 82 new + 469 unchanged - 30 fixed = 551 total
(was 499) |
| +1 :green_heart: | mvnsite | 4m 39s | | the patch passed |
| -1 :x: | javadoc | 0m 55s |
[/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/4/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 2 new + 97 unchanged
- 2 fixed = 99 total (was 99) |
| -1 :x: | javadoc | 1m 30s |
[/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/4/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt)
|
hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 3 new
+ 0 unchanged - 0 fixed = 3 total (was 0) |
| -1 :x: | javadoc | 0m 36s |
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=634472=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-634472
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 11:55
Start Date: 05/Aug/21 11:55
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2735:
URL: https://github.com/apache/hadoop/pull/2735#issuecomment-892760289
I have stopped working on this. Feel free to take it up
I originally thought "hey, we could just make this public and there'd be a
good rename", but as usual the challenge becomes one of strictly implementing
the preconditions. FileContext does that, though non-atomically; Factoring out
all that policy at least makes things consistent.
But, having dealt with other rename related trouble recently, I'm thinking
really I'd want a new builder-based rename
```
Future foutcome = FS.renamePath(source, dest)
.must("fs.opt.rename.atomic", true)
.build()
RenameOutcome outcome = outcome.get();
class RenameOutcome implements IOStatisticsSource {
}
Why this?
* Allows for stores which count IO Costs of renames to report them
* You need some kind of return type for java futures
* builder options would let you say whether you MUST have atomic rename, in
which case
-no s3a, wasb or gcs rename for you.
Why async?
* so slow stores can be obviously slow about it
* let you pass in a progressable. Distcp could do this so stop tasks failing
during rename of big files on non -direct uploads to s3; same for
FileOutputCommitter
```
Future foutcome = FS.renamePath(source, dest)
.withProgress(reporter)
.build()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 634472)
Time Spent: 4h 20m (was: 4h 10m)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=633657=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633657
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 15:35
Start Date: 04/Aug/21 15:35
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2735:
URL: https://github.com/apache/hadoop/pull/2735#issuecomment-892760289
I have stopped working on this. Feel free to take it up
I originally thought "hey, we could just make this public and there'd be a
good rename", but as usual the challenge becomes one of strictly implementing
the preconditions. FileContext does that, though non-atomically; Factoring out
all that policy at least makes things consistent.
But, having dealt with other rename related trouble recently, I'm thinking
really I'd want a new builder-based rename
```
Future foutcome = FS.renamePath(source, dest)
.must("fs.opt.rename.atomic", true)
.build()
RenameOutcome outcome = outcome.get();
class RenameOutcome implements IOStatisticsSource {
}
Why this?
* Allows for stores which count IO Costs of renames to report them
* You need some kind of return type for java futures
* builder options would let you say whether you MUST have atomic rename, in
which case
-no s3a, wasb or gcs rename for you.
Why async?
* so slow stores can be obviously slow about it
* let you pass in a progressable. Distcp could do this so stop tasks failing
during rename of big files on non -direct uploads to s3; same for
FileOutputCommitter
```
Future foutcome = FS.renamePath(source, dest)
.withProgress(reporter)
.build()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633657)
Time Spent: 4h 10m (was: 4h)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=622643=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-622643
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 14/Jul/21 18:39
Start Date: 14/Jul/21 18:39
Worklog Time Spent: 10m
Work Description: jteagles commented on pull request #2735:
URL: https://github.com/apache/hadoop/pull/2735#issuecomment-880122931
@steveloughran, is this still active. I'm interested in using this
functionality and would be willing to help review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 622643)
Time Spent: 4h (was: 3h 50m)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=588857=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-588857
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 26/Apr/21 06:54
Start Date: 26/Apr/21 06:54
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2735:
URL: https://github.com/apache/hadoop/pull/2735#issuecomment-826510805
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 21m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 9 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 50s | | trunk passed |
| +1 :green_heart: | compile | 22m 15s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 18m 59s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 8s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 20s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 9m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 30s | | the patch passed |
| +1 :green_heart: | compile | 21m 35s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 35s | |
root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 0 new + 1926 unchanged - 5
fixed = 1926 total (was 1931) |
| +1 :green_heart: | compile | 19m 1s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 19m 1s | |
root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 0 new + 1825 unchanged -
5 fixed = 1825 total (was 1830) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/4/artifact/out/blanks-eol.txt)
| The patch has 7 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 59s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/4/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 82 new + 469 unchanged - 30 fixed = 551 total
(was 499) |
| +1 :green_heart: | mvnsite | 4m 39s | | the patch passed |
| -1 :x: | javadoc | 0m 55s |
[/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/4/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 2 new + 97 unchanged
- 2 fixed = 99 total (was 99) |
| -1 :x: | javadoc | 1m 30s |
[/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/4/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt)
|
hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 3 new
+ 0 unchanged - 0 fixed = 3 total (was 0) |
| -1 :x: | javadoc | 0m 36s |
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=588691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-588691
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 26/Apr/21 05:22
Start Date: 26/Apr/21 05:22
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2735:
URL: https://github.com/apache/hadoop/pull/2735#issuecomment-826510805
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 21m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 9 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 50s | | trunk passed |
| +1 :green_heart: | compile | 22m 15s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 18m 59s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 8s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 20s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 9m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 30s | | the patch passed |
| +1 :green_heart: | compile | 21m 35s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 35s | |
root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 0 new + 1926 unchanged - 5
fixed = 1926 total (was 1931) |
| +1 :green_heart: | compile | 19m 1s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 19m 1s | |
root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 0 new + 1825 unchanged -
5 fixed = 1825 total (was 1830) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/4/artifact/out/blanks-eol.txt)
| The patch has 7 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 59s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/4/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 82 new + 469 unchanged - 30 fixed = 551 total
(was 499) |
| +1 :green_heart: | mvnsite | 4m 39s | | the patch passed |
| -1 :x: | javadoc | 0m 55s |
[/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/4/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 2 new + 97 unchanged
- 2 fixed = 99 total (was 99) |
| -1 :x: | javadoc | 1m 30s |
[/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/4/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt)
|
hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 3 new
+ 0 unchanged - 0 fixed = 3 total (was 0) |
| -1 :x: | javadoc | 0m 36s |
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=560013=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-560013
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 02/Mar/21 15:56
Start Date: 02/Mar/21 15:56
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2735:
URL: https://github.com/apache/hadoop/pull/2735#issuecomment-789011590
Quick review of this, especially the factoring out
Good:
* unified logic in one place
Bad
* it's still complex
* it's not easy for S3A to optimise further
Looking at s3a rename I now want
* getFileStatus on source file to return the metadata so that there's no
second call on the copy
* probe for parent dir to know it expects a dir. e.g callback
`getDirStatus(dir)`.
* if the source is a dir, to initiate a LIST and have that feed straight
into the object scan. That's tricky, but you know...
* option to turn off the check for the dir existing.
* async fetch of all the probes
so: all the callbacks would be completable futures; default impl would be
sequential, but for the stores we can go async.
+ split into API (o.a.h.fs.api) and impl and have something in API To be the
factory for the impl: downstream classes do not need to look into .impl.
No timetable for this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 560013)
Time Spent: 3.5h (was: 3h 20m)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[ https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=560006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-560006 ] ASF GitHub Bot logged work on HADOOP-11452: --- Author: ASF GitHub Bot Created on: 02/Mar/21 15:29 Start Date: 02/Mar/21 15:29 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2735: URL: https://github.com/apache/hadoop/pull/2735#issuecomment-788992374 ## full commit logs of squashed commits - PR with the previous patches. This is a WiP but ready for some iniital review; lacks tests & spec. Also, because the base FileSystem.rename/3 does its own src/dest checks, it's less efficient against object stores. They need their own high-perf subclass. Change-Id: I1586ef2290d7a3d2d33b1a32e2f0999b07c26143 HADOOP-11452 rename/3 has tests for local fs, raw local and hdfs. + updates docs + contains fix for HADOOP-16255 : checksum FS doesn't do rename/3 properly tests are based on those of rename; there's still some stuff about renaming into an empty directory which makes me suspect there's ambiguity there Change-Id: Ic1ca6cbe3e9ca80ab8e1459167a7012678e856fc HADOOP-11452 rename(path, path, options) to become public * exceptions match expectations of FSMainOperationsBaseTest * clean up FSMainOperationsBaseTest by moving to intercept, try with resources. * S3A to implement subclass of FSMainOperationsBaseTest, ITestS3AFSMainOperations * S3A to implement ITestS3AContractRenameEx * Add protected override point, executeInnerRename, for implementing rename/3; base class calls rename() and throws if that returns false, i.e. current behaviour * S3A overrides executeInnerRename to call its own innerRename() and so raise all failures as IOEs. Issues: is the rename point executeInnerRename() a good name? S3AFS should really implement a direct rename/3 so there's no duplication of checks for parent etc, maybe even passing the values down. Or we make sure innerRename() is consistent with the spec, which primarily means logic about dest dir existing. Change-Id: I0ac3695434d85072ab860854e5e88bc6d36e754a HADOOP-11452 trying to move rename/3 logic into its own class Change-Id: If2ab67152e08e4c2a225f6e89c24a5d1ff79ee59 HADOOP-15183: S3AFileSystem does rename/3 Factored the rename check logic out into a RenameHelper which is then used in S3A FileSystem as the PoC of how it can use the RenameHelper then directly invoke the inner operations. Added some more @Retry attributes as appropriate. Conclusion: it works, but for efficient IOPS then `innerRename()` needs to take optional source and dest status values and so so omit repeating the checks. For more work on this * the tests; file context has some so review and add to AbstractContractRenameTest. Also, based on some feedback from Sean Mackrory: verify the renamed files actually have the data. * move internal use of rename (distcp, maprv2 FileOutputcommitter), etc to use this. of the 50+ places which call rename, they seem split 3-ways int 1. subclasses and forwarding 2. invocations whch check the return value and throw an IOE 3. invocations which are written on the assumption that renames raise exceptions on failure 2 & 3 are the ones to change. Change-Id: Id77ed7352b9d5ddb124f9191c5c5f1b8a76da7bb HADOOP-11452. Rename Review of RenameHelper based on current coding styles and plans (IOStats, etc) Change-Id: I3d39ee3ed04a10e7db2c2b2c79833b945b4d691b HADOOP-11452 Rename/3 S3A high performance edition. This avoids all surplus S3 calls and has meaningful exception raising. TODO: * pull the S3A code out into is own operation + extra callbacks (innerGetFileStatus is all that's needed) * see if the FileContext default logic can be pulled out too, using a custom set of callbacks. If it can't the logic is broken. * do some testing Change-Id: I408b2cfe93f266cf0c9084fa8f05bb84b65c2bad HADOOP-11452 Rename/3 * Add RawLocalFileSystem rename with useful errors * pull out all rename docs into their own filesystem.md doc * Add a callback impl which => FileContext too, at least for the nonatomic rename. FC doesn't do path name checking. Should we? Proposed changes * move the new interfaces up to o.a.h.fs, so that .impl is never imported in FileSystem APIS. * remove the createRename callbacks method, just have stores with implement rename/3 other than the base FS to override all of rename 3. Change-Id: I1fab598553b8e9de4d659b80248bac440dbac018 This is an automated
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=560005=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-560005
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 02/Mar/21 15:28
Start Date: 02/Mar/21 15:28
Worklog Time Spent: 10m
Work Description: steveloughran opened a new pull request #2735:
URL: https://github.com/apache/hadoop/pull/2735
make rename/3 public. Rollup of #743 commits rebased to trunk
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 560005)
Time Spent: 3h 10m (was: 3h)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=560001=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-560001
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 02/Mar/21 15:24
Start Date: 02/Mar/21 15:24
Worklog Time Spent: 10m
Work Description: steveloughran closed pull request #743:
URL: https://github.com/apache/hadoop/pull/743
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 560001)
Time Spent: 3h (was: 2h 50m)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=56=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-56
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 02/Mar/21 15:24
Start Date: 02/Mar/21 15:24
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #743:
URL: https://github.com/apache/hadoop/pull/743#issuecomment-788988980
closing and creating a new rebased/squashed PR. Still need to look at the
comments related to permissions
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 56)
Time Spent: 2h 50m (was: 2h 40m)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[ https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=487733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487733 ] ASF GitHub Bot logged work on HADOOP-11452: --- Author: ASF GitHub Bot Created on: 22/Sep/20 03:31 Start Date: 22/Sep/20 03:31 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #743: URL: https://github.com/apache/hadoop/pull/743#issuecomment-689849388 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 30m 11s | trunk passed | | +1 :green_heart: | compile | 23m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | compile | 0m 35s | root in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 22s | The patch fails to run checkstyle in root | | -1 :x: | mvnsite | 0m 29s | hadoop-common in trunk failed. | | -1 :x: | mvnsite | 0m 34s | hadoop-hdfs in trunk failed. | | -1 :x: | mvnsite | 0m 29s | hadoop-hdfs-client in trunk failed. | | -1 :x: | mvnsite | 0m 35s | hadoop-aws in trunk failed. | | -1 :x: | shadedclient | 12m 52s | branch has errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 4m 6s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 5m 41s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 12m 7s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 23s | the patch passed | | -1 :x: | compile | 15m 11s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javac | 15m 11s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 27s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | javac | 0m 27s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 3m 8s | root: The patch generated 587 new + 0 unchanged - 0 fixed = 587 total (was 0) | | +1 :green_heart: | mvnsite | 4m 43s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedclient | 14m 15s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 31s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javadoc | 1m 20s | hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) | | -1 :x: | findbugs | 2m 15s | hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | -1 :x: | unit | 9m 24s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 3s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 96m 17s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 0m 33s | hadoop-openstack in the patch passed. | | +1 :green_heart: | unit | 1m 24s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 48s | The patch does not generate ASF License warnings. | | | | 267m 36s | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-common-project/hadoop-common | | | Should org.apache.hadoop.fs.impl.RenameHelper$RenameValidationResult be a _static_ inner class? At RenameHelper.java:inner
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[ https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=487203=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-487203 ] ASF GitHub Bot logged work on HADOOP-11452: --- Author: ASF GitHub Bot Created on: 21/Sep/20 18:56 Start Date: 21/Sep/20 18:56 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #743: URL: https://github.com/apache/hadoop/pull/743#issuecomment-689849388 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 30m 11s | trunk passed | | +1 :green_heart: | compile | 23m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | compile | 0m 35s | root in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 22s | The patch fails to run checkstyle in root | | -1 :x: | mvnsite | 0m 29s | hadoop-common in trunk failed. | | -1 :x: | mvnsite | 0m 34s | hadoop-hdfs in trunk failed. | | -1 :x: | mvnsite | 0m 29s | hadoop-hdfs-client in trunk failed. | | -1 :x: | mvnsite | 0m 35s | hadoop-aws in trunk failed. | | -1 :x: | shadedclient | 12m 52s | branch has errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 4m 6s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 5m 41s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 12m 7s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 23s | the patch passed | | -1 :x: | compile | 15m 11s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javac | 15m 11s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 27s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | javac | 0m 27s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 3m 8s | root: The patch generated 587 new + 0 unchanged - 0 fixed = 587 total (was 0) | | +1 :green_heart: | mvnsite | 4m 43s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedclient | 14m 15s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 31s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javadoc | 1m 20s | hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) | | -1 :x: | findbugs | 2m 15s | hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | -1 :x: | unit | 9m 24s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 3s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 96m 17s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 0m 33s | hadoop-openstack in the patch passed. | | +1 :green_heart: | unit | 1m 24s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 48s | The patch does not generate ASF License warnings. | | | | 267m 36s | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-common-project/hadoop-common | | | Should org.apache.hadoop.fs.impl.RenameHelper$RenameValidationResult be a _static_ inner class? At RenameHelper.java:inner
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=483957=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-483957
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 14/Sep/20 12:08
Start Date: 14/Sep/20 12:08
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #743:
URL: https://github.com/apache/hadoop/pull/743#issuecomment-692010404
as per Daryn's comments in :
https://issues.apache.org/jira/browse/HADOOP-16281?focusedCommentId=16832628=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16832628
No changes to HDFS error messages. Review this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 483957)
Time Spent: 2h 20m (was: 2h 10m)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[ https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=481375=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481375 ] ASF GitHub Bot logged work on HADOOP-11452: --- Author: ASF GitHub Bot Created on: 10/Sep/20 11:16 Start Date: 10/Sep/20 11:16 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #743: URL: https://github.com/apache/hadoop/pull/743#issuecomment-685019888 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 28m 46s | trunk passed | | +1 :green_heart: | compile | 20m 47s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 17m 39s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 1s | trunk passed | | +1 :green_heart: | mvnsite | 5m 5s | trunk passed | | +1 :green_heart: | shadedclient | 24m 24s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 6s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 37s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 45s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 9m 54s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 39s | the patch passed | | +1 :green_heart: | compile | 20m 14s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javac | 20m 14s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 1 new + 2050 unchanged - 5 fixed = 2051 total (was 2055) | | +1 :green_heart: | compile | 17m 40s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | javac | 17m 40s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 1944 unchanged - 5 fixed = 1945 total (was 1949) | | -0 :warning: | checkstyle | 3m 3s | root: The patch generated 103 new + 484 unchanged - 30 fixed = 587 total (was 514) | | +1 :green_heart: | mvnsite | 5m 0s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedclient | 15m 43s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 7s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javadoc | 1m 30s | hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) | | -1 :x: | findbugs | 2m 22s | hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | -1 :x: | unit | 9m 48s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 7s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 126m 42s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 0m 38s | hadoop-openstack in the patch passed. | | +1 :green_heart: | unit | 1m 58s | hadoop-aws in the patch passed. | | -1 :x: | asflicense | 0m 59s | The patch generated 3 ASF License warnings. | | | | 345m 9s | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-common-project/hadoop-common | | | Should org.apache.hadoop.fs.impl.RenameHelper$RenameValidationResult be a _static_ inner class? At RenameHelper.java:inner class? At RenameHelper.java:[line 320] | |
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[ https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=481098=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-481098 ] ASF GitHub Bot logged work on HADOOP-11452: --- Author: ASF GitHub Bot Created on: 09/Sep/20 22:12 Start Date: 09/Sep/20 22:12 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #743: URL: https://github.com/apache/hadoop/pull/743#issuecomment-689849388 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 36s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 30m 11s | trunk passed | | +1 :green_heart: | compile | 23m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | compile | 0m 35s | root in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 0m 22s | The patch fails to run checkstyle in root | | -1 :x: | mvnsite | 0m 29s | hadoop-common in trunk failed. | | -1 :x: | mvnsite | 0m 34s | hadoop-hdfs in trunk failed. | | -1 :x: | mvnsite | 0m 29s | hadoop-hdfs-client in trunk failed. | | -1 :x: | mvnsite | 0m 35s | hadoop-aws in trunk failed. | | -1 :x: | shadedclient | 12m 52s | branch has errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 4m 6s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 5m 41s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 12m 7s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 23s | the patch passed | | -1 :x: | compile | 15m 11s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javac | 15m 11s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 0m 27s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | javac | 0m 27s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 3m 8s | root: The patch generated 587 new + 0 unchanged - 0 fixed = 587 total (was 0) | | +1 :green_heart: | mvnsite | 4m 43s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedclient | 14m 15s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 2m 31s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javadoc | 1m 20s | hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) | | -1 :x: | findbugs | 2m 15s | hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | -1 :x: | unit | 9m 24s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 3s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 96m 17s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 0m 33s | hadoop-openstack in the patch passed. | | +1 :green_heart: | unit | 1m 24s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 48s | The patch does not generate ASF License warnings. | | | | 267m 36s | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-common-project/hadoop-common | | | Should org.apache.hadoop.fs.impl.RenameHelper$RenameValidationResult be a _static_ inner class? At RenameHelper.java:inner class? At
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[ https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=477363=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-477363 ] ASF GitHub Bot logged work on HADOOP-11452: --- Author: ASF GitHub Bot Created on: 01/Sep/20 17:33 Start Date: 01/Sep/20 17:33 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #743: URL: https://github.com/apache/hadoop/pull/743#issuecomment-685019888 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 28m 46s | trunk passed | | +1 :green_heart: | compile | 20m 47s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 17m 39s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 1s | trunk passed | | +1 :green_heart: | mvnsite | 5m 5s | trunk passed | | +1 :green_heart: | shadedclient | 24m 24s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 6s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 37s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 45s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 9m 54s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 39s | the patch passed | | +1 :green_heart: | compile | 20m 14s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javac | 20m 14s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 1 new + 2050 unchanged - 5 fixed = 2051 total (was 2055) | | +1 :green_heart: | compile | 17m 40s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | javac | 17m 40s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 1944 unchanged - 5 fixed = 1945 total (was 1949) | | -0 :warning: | checkstyle | 3m 3s | root: The patch generated 103 new + 484 unchanged - 30 fixed = 587 total (was 514) | | +1 :green_heart: | mvnsite | 5m 0s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedclient | 15m 43s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 7s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javadoc | 1m 30s | hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) | | -1 :x: | findbugs | 2m 22s | hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | -1 :x: | unit | 9m 48s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 7s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 126m 42s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 0m 38s | hadoop-openstack in the patch passed. | | +1 :green_heart: | unit | 1m 58s | hadoop-aws in the patch passed. | | -1 :x: | asflicense | 0m 59s | The patch generated 3 ASF License warnings. | | | | 345m 9s | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-common-project/hadoop-common | | | Should org.apache.hadoop.fs.impl.RenameHelper$RenameValidationResult be a _static_ inner class? At RenameHelper.java:inner class? At RenameHelper.java:[line 320] | | Failed
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=477179=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-477179
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 01/Sep/20 12:06
Start Date: 01/Sep/20 12:06
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #743:
URL: https://github.com/apache/hadoop/pull/743#issuecomment-683328922
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 477179)
Time Spent: 1h 40m (was: 1.5h)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[ https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=476273=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476273 ] ASF GitHub Bot logged work on HADOOP-11452: --- Author: ASF GitHub Bot Created on: 30/Aug/20 18:06 Start Date: 30/Aug/20 18:06 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #743: URL: https://github.com/apache/hadoop/pull/743#issuecomment-683451503 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 25m 46s | trunk passed | | +1 :green_heart: | compile | 19m 27s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 16m 46s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 2m 55s | trunk passed | | +1 :green_heart: | mvnsite | 5m 38s | trunk passed | | +1 :green_heart: | shadedclient | 23m 31s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 31s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 4m 55s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 49s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 9m 57s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 52s | the patch passed | | +1 :green_heart: | compile | 19m 47s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javac | 19m 47s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 1 new + 2050 unchanged - 5 fixed = 2051 total (was 2055) | | +1 :green_heart: | compile | 18m 21s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | javac | 18m 21s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 1944 unchanged - 5 fixed = 1945 total (was 1949) | | -0 :warning: | checkstyle | 2m 50s | root: The patch generated 65 new + 456 unchanged - 31 fixed = 521 total (was 487) | | +1 :green_heart: | mvnsite | 5m 27s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedclient | 14m 30s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javadoc | 1m 31s | hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | -1 :x: | findbugs | 2m 28s | hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | -1 :x: | unit | 9m 27s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 21s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 99m 38s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 0m 48s | hadoop-openstack in the patch passed. | | +1 :green_heart: | unit | 1m 42s | hadoop-aws in the patch passed. | | -1 :x: | asflicense | 1m 8s | The patch generated 3 ASF License warnings. | | | | 313m 14s | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-common-project/hadoop-common | | | Should org.apache.hadoop.fs.impl.RenameHelper$RenameValidationResult be a _static_ inner class? At RenameHelper.java:inner class? At RenameHelper.java:[line 320] | | Failed
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=476154=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476154
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 29/Aug/20 19:20
Start Date: 29/Aug/20 19:20
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#743:
URL: https://github.com/apache/hadoop/pull/743#discussion_r479681262
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSExceptionMessages.java
##
@@ -51,4 +51,86 @@
public static final String PERMISSION_DENIED_BY_STICKY_BIT =
"Permission denied by sticky bit";
+
+ /**
+ * Renaming a destination under source is forbidden.
+ * This is a format string.
+ */
+ public static final String RENAME_DEST_UNDER_SOURCE =
+ "Rename destination %s is a directory or file under source %s";
+
+ /**
+ * Renaming a destination to source is forbidden.
+ * This is a format string.
+ */
+ public static final String RENAME_DEST_EQUALS_SOURCE =
+ "The source %s and destination %s are the same";
+
+ /**
+ * Renaming to root is forbidden.
+ */
+ public static final String RENAME_DEST_IS_ROOT =
+ "Rename destination cannot be the root";
+
+ /**
+ * The parent of a rename destination is not found.
+ * This is a format string.
+ */
+ public static final String RENAME_DEST_NO_PARENT_OF =
+ "Rename destination parent of %s not found";
+
+ /**
+ * The parent of a rename destination is not found.
+ * This is a format string, taking the parent path of the destination
+ */
+ public static final String RENAME_DEST_NO_PARENT =
+ "Rename destination parent %s not found";
+
+ /**
+ * The parent of a rename destination is not a directory.
+ * This is a format string.
+ */
+ public static final String RENAME_DEST_PARENT_NOT_DIRECTORY =
+ "Rename destination parent %s is a file";
+
+ /**
+ * The rename destination is not an empty directory.
+ * This is a format string.
+ */
+ public static final String RENAME_DEST_NOT_EMPTY =
+ "Rename destination directory is not empty: %s";
+
+ /**
+ * The rename destination is not an empty directory.
+ * This is a format string.
+ */
+ public static final String RENAME_DEST_EXISTS =
+ "Rename destination %s already exists";
+
+ /**
+ * The rename source doesn't exist.
+ * This is a format string.
+ */
+ public static final String RENAME_SOURCE_NOT_FOUND =
+ "Rename source %s is not found";
+
+ /**
+ * The rename source and dest are off different types
Review comment:
fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 476154)
Time Spent: 1h 10m (was: 1h)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=476155=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476155
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 29/Aug/20 19:20
Start Date: 29/Aug/20 19:20
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#743:
URL: https://github.com/apache/hadoop/pull/743#discussion_r479681352
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/RenameHelper.java
##
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Optional;
+
+import org.slf4j.Logger;
+
+import org.apache.hadoop.fs.FileAlreadyExistsException;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Options;
+import org.apache.hadoop.fs.ParentNotDirectoryException;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+
+import static
org.apache.hadoop.fs.FSExceptionMessages.RENAME_DEST_EQUALS_SOURCE;
Review comment:
done!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 476155)
Time Spent: 1h 20m (was: 1h 10m)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=476151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476151
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 29/Aug/20 19:15
Start Date: 29/Aug/20 19:15
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#743:
URL: https://github.com/apache/hadoop/pull/743#discussion_r479680825
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java
##
@@ -670,6 +670,27 @@ public boolean rename(Path src, Path dst) throws
IOException {
}
}
+ /**
+ * Rename files/dirs.
+ */
+ @Override
+ @SuppressWarnings("deprecation")
+ public void rename(final Path src,
+ Path dst,
+ final Options.Rename... options)
+ throws IOException {
+ fs.rename(src, dst, options);
+
+ Path srcCheckFile = getChecksumFile(src);
+ Path dstCheckFile = getChecksumFile(dst);
+ if (fs.exists(srcCheckFile)) { //try to rename checksum
+fs.rename(srcCheckFile, dstCheckFile, Options.Rename.OVERWRITE);
Review comment:
+added comments in Option as well as Checksum FS
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 476151)
Time Spent: 1h (was: 50m)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=476149=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476149
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 29/Aug/20 19:01
Start Date: 29/Aug/20 19:01
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#743:
URL: https://github.com/apache/hadoop/pull/743#discussion_r479679604
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java
##
@@ -670,6 +670,27 @@ public boolean rename(Path src, Path dst) throws
IOException {
}
}
+ /**
+ * Rename files/dirs.
+ */
+ @Override
+ @SuppressWarnings("deprecation")
+ public void rename(final Path src,
+ Path dst,
+ final Options.Rename... options)
+ throws IOException {
+ fs.rename(src, dst, options);
+
+ Path srcCheckFile = getChecksumFile(src);
+ Path dstCheckFile = getChecksumFile(dst);
+ if (fs.exists(srcCheckFile)) { //try to rename checksum
+fs.rename(srcCheckFile, dstCheckFile, Options.Rename.OVERWRITE);
Review comment:
interesting thought. The only current option is trash; don't know if we
should retain that. I will add a comment in the javadocs of Options.Rename on
this topic
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 476149)
Time Spent: 50m (was: 40m)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=476148=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476148
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 29/Aug/20 18:55
Start Date: 29/Aug/20 18:55
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#743:
URL: https://github.com/apache/hadoop/pull/743#discussion_r479679077
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java
##
@@ -670,6 +670,27 @@ public boolean rename(Path src, Path dst) throws
IOException {
}
}
+ /**
+ * Rename files/dirs.
+ */
+ @Override
+ @SuppressWarnings("deprecation")
+ public void rename(final Path src,
+ Path dst,
+ final Options.Rename... options)
+ throws IOException {
+ fs.rename(src, dst, options);
Review comment:
if options = NONE and the dest exists, the call MUST raise an exception.
I've just reviewed this carefully and added comments to clarify
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 476148)
Time Spent: 40m (was: 0.5h)
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[ https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=476146=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476146 ] ASF GitHub Bot logged work on HADOOP-11452: --- Author: ASF GitHub Bot Created on: 29/Aug/20 18:51 Start Date: 29/Aug/20 18:51 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #743: URL: https://github.com/apache/hadoop/pull/743#issuecomment-683328922 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 9 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 25m 55s | trunk passed | | +1 :green_heart: | compile | 19m 31s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 16m 48s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 2m 54s | trunk passed | | +1 :green_heart: | mvnsite | 5m 38s | trunk passed | | +1 :green_heart: | shadedclient | 23m 13s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 48s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 5m 19s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 0m 54s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 10m 1s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 44s | the patch passed | | +1 :green_heart: | compile | 18m 50s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javac | 18m 50s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 1 new + 2050 unchanged - 5 fixed = 2051 total (was 2055) | | +1 :green_heart: | compile | 17m 1s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | -1 :x: | javac | 17m 1s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 1944 unchanged - 5 fixed = 1945 total (was 1949) | | -0 :warning: | checkstyle | 2m 48s | root: The patch generated 63 new + 292 unchanged - 48 fixed = 355 total (was 340) | | +1 :green_heart: | mvnsite | 5m 38s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedclient | 14m 22s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 45s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | -1 :x: | javadoc | 1m 34s | hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | | -1 :x: | findbugs | 2m 22s | hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | -1 :x: | unit | 9m 29s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 22s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 97m 40s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | unit | 0m 46s | hadoop-openstack in the patch passed. | | +1 :green_heart: | unit | 1m 44s | hadoop-aws in the patch passed. | | -1 :x: | asflicense | 1m 8s | The patch generated 3 ASF License warnings. | | | | 310m 9s | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-common-project/hadoop-common | | | Should org.apache.hadoop.fs.impl.RenameHelper$RenameValidationResult be a _static_ inner class? At RenameHelper.java:inner class? At RenameHelper.java:[line 320] | | Failed
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=476144=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476144
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 29/Aug/20 18:39
Start Date: 29/Aug/20 18:39
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#743:
URL: https://github.com/apache/hadoop/pull/743#discussion_r479677745
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java
##
@@ -195,99 +214,82 @@ public void testWDAbsolute() throws IOException {
Path absoluteDir = new Path(fSys.getUri() + "/test/existingDir");
fSys.mkdirs(absoluteDir);
fSys.setWorkingDirectory(absoluteDir);
-Assert.assertEquals(absoluteDir, fSys.getWorkingDirectory());
+assertEquals(absoluteDir, fSys.getWorkingDirectory());
}
@Test
public void testMkdirs() throws Exception {
Path testDir = getTestRootPath(fSys, "test/hadoop");
-Assert.assertFalse(exists(fSys, testDir));
-Assert.assertFalse(isFile(fSys, testDir));
+assertFalse(exists(fSys, testDir));
+assertFalse(isFile(fSys, testDir));
fSys.mkdirs(testDir);
-Assert.assertTrue(exists(fSys, testDir));
-Assert.assertFalse(isFile(fSys, testDir));
+assertTrue(exists(fSys, testDir));
+assertFalse(isFile(fSys, testDir));
fSys.mkdirs(testDir);
-Assert.assertTrue(exists(fSys, testDir));
-Assert.assertFalse(isFile(fSys, testDir));
+assertTrue(exists(fSys, testDir));
+assertFalse(isFile(fSys, testDir));
Path parentDir = testDir.getParent();
-Assert.assertTrue(exists(fSys, parentDir));
-Assert.assertFalse(isFile(fSys, parentDir));
+assertTrue(exists(fSys, parentDir));
+assertFalse(isFile(fSys, parentDir));
Path grandparentDir = parentDir.getParent();
-Assert.assertTrue(exists(fSys, grandparentDir));
-Assert.assertFalse(isFile(fSys, grandparentDir));
+assertTrue(exists(fSys, grandparentDir));
+assertFalse(isFile(fSys, grandparentDir));
}
@Test
public void testMkdirsFailsForSubdirectoryOfExistingFile() throws Exception {
Path testDir = getTestRootPath(fSys, "test/hadoop");
-Assert.assertFalse(exists(fSys, testDir));
+assertFalse(exists(fSys, testDir));
fSys.mkdirs(testDir);
-Assert.assertTrue(exists(fSys, testDir));
+assertTrue(exists(fSys, testDir));
createFile(getTestRootPath(fSys, "test/hadoop/file"));
Path testSubDir = getTestRootPath(fSys, "test/hadoop/file/subdir");
-try {
- fSys.mkdirs(testSubDir);
- Assert.fail("Should throw IOException.");
-} catch (IOException e) {
- // expected
-}
-Assert.assertFalse(exists(fSys, testSubDir));
+intercept(IOException.class, () ->
+ fSys.mkdirs(testSubDir));
+assertFalse(exists(fSys, testSubDir));
Path testDeepSubDir = getTestRootPath(fSys,
"test/hadoop/file/deep/sub/dir");
-try {
- fSys.mkdirs(testDeepSubDir);
- Assert.fail("Should throw IOException.");
-} catch (IOException e) {
- // expected
-}
-Assert.assertFalse(exists(fSys, testDeepSubDir));
+intercept(IOException.class, () ->
+fSys.mkdirs(testDeepSubDir));
+assertFalse(exists(fSys, testDeepSubDir));
}
@Test
- public void testGetFileStatusThrowsExceptionForNonExistentFile()
-throws Exception {
-try {
- fSys.getFileStatus(getTestRootPath(fSys, "test/hadoop/file"));
- Assert.fail("Should throw FileNotFoundException");
-} catch (FileNotFoundException e) {
- // expected
-}
- }
+ public void testGetFileStatusThrowsExceptionForNonExistentFile()
+ throws Exception {
+intercept(FileNotFoundException.class, () ->
+ fSys.getFileStatus(getTestRootPath(fSys, "test/hadoop/file")));
+ }
@Test
public void testListStatusThrowsExceptionForNonExistentFile()
- throws Exception {
-try {
- fSys.listStatus(getTestRootPath(fSys, "test/hadoop/file"));
- Assert.fail("Should throw FileNotFoundException");
-} catch (FileNotFoundException fnfe) {
- // expected
-}
+ throws Exception {
+intercept(FileNotFoundException.class, () ->
+ fSys.listStatus(getTestRootPath(fSys, "test/hadoop/file")));
}
@Test
public void testListStatusThrowsExceptionForUnreadableDir()
throws Exception {
+assumePermissionsSupported();
Path testRootDir = getTestRootPath(fSys, "test/hadoop/dir");
Path obscuredDir = new Path(testRootDir, "foo");
Path subDir = new Path(obscuredDir, "bar"); //so foo is non-empty
fSys.mkdirs(subDir);
fSys.setPermission(obscuredDir, new FsPermission((short)0)); //no access
try {
- fSys.listStatus(obscuredDir);
-
[jira] [Work logged] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?focusedWorklogId=476143=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476143
]
ASF GitHub Bot logged work on HADOOP-11452:
---
Author: ASF GitHub Bot
Created on: 29/Aug/20 18:28
Start Date: 29/Aug/20 18:28
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#743:
URL: https://github.com/apache/hadoop/pull/743#discussion_r479676808
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java
##
@@ -623,26 +625,25 @@ public void
testGlobStatusFilterWithMultiplePathWildcardsAndNonTrivialFilter()
FileStatus[] filteredPaths = fSys.globStatus(
getTestRootPath(fSys, "test/hadoop/a??"),
TEST_X_FILTER);
-Assert.assertEquals(2, filteredPaths.length);
-Assert.assertTrue(containsTestRootPath(getTestRootPath(fSys, TEST_DIR_AXA),
+assertEquals(2, filteredPaths.length);
+assertTrue(containsTestRootPath(getTestRootPath(fSys, TEST_DIR_AXA),
filteredPaths));
-Assert.assertTrue(containsTestRootPath(getTestRootPath(fSys, TEST_DIR_AXX),
+assertTrue(containsTestRootPath(getTestRootPath(fSys, TEST_DIR_AXX),
filteredPaths));
}
@Test
public void testGlobStatusThrowsExceptionForUnreadableDir()
throws Exception {
+assumePermissionsSupported();
Path testRootDir = getTestRootPath(fSys, "test/hadoop/dir");
Path obscuredDir = new Path(testRootDir, "foo");
Path subDir = new Path(obscuredDir, "bar"); //so foo is non-empty
fSys.mkdirs(subDir);
fSys.setPermission(obscuredDir, new FsPermission((short)0)); //no access
try {
- fSys.globStatus(getTestRootPath(fSys, "test/hadoop/dir/foo/*"));
- Assert.fail("Should throw IOException");
-} catch (IOException ioe) {
- // expected
+ intercept(IOException.class, () ->
Review comment:
I'm going to revert all changes to this test suite other than those
which go near rename, so whatever is tested for today is going to be exactly
what is continued to be tested for. I Don't know what the nuances of HDFS
permissions are.
I'd assume that the getFileStatus call only needs metadata/list permissions,
rather than read access. That HDFS doc states that client needs execute
permission on all parent directories, so that it can get from / to the file.
Therefore the "has permissions to call getFileXXX" is implicit if you can get
to the dir, right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 476143)
Remaining Estimate: 0h
Time Spent: 10m
> Make FileSystem.rename(path, path, options) public, specified, tested
> -
>
> Key: HADOOP-11452
> URL: https://issues.apache.org/jira/browse/HADOOP-11452
> Project: Hadoop Common
> Issue Type: Task
> Components: fs
>Affects Versions: 2.7.3
>Reporter: Yi Liu
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-11452-001.patch, HADOOP-11452-002.patch,
> HADOOP-14452-004.patch, HADOOP-14452-branch-2-003.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently in {{FileSystem}}, {{rename}} with _Rename options_ is protected
> and with _deprecated_ annotation. And the default implementation is not
> atomic.
> So this method is not able to be used outside. On the other hand, HDFS has a
> good and atomic implementation. (Also an interesting thing in {{DFSClient}},
> the _deprecated_ annotations for these two methods are opposite).
> It makes sense to make public for {{rename}} with _Rename options_, since
> it's atomic for rename+overwrite, also it saves RPC calls if user desires
> rename+overwrite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
