[jira] [Created] (HADOOP-16687) ABFS: Fix testcase added for HADOOP-16138 for namespace enabled account
Sneha Vijayarajan created HADOOP-16687: -- Summary: ABFS: Fix testcase added for HADOOP-16138 for namespace enabled account Key: HADOOP-16687 URL: https://issues.apache.org/jira/browse/HADOOP-16687 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Reporter: Sneha Vijayarajan Assignee: Sneha Vijayarajan Fix For: 3.3.0 ExponentialRetryPolicy will retry requests 30 times which is the default retry count in code whe HttpRequests fail. If the client too has re-try handling, the process seems hanged to the client App due to the high number of retries. This Jira aims to provide a config control for retry count. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16677) Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout.
[
https://issues.apache.org/jira/browse/HADOOP-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968073#comment-16968073
]
Hadoop QA commented on HADOOP-16677:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
11s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
58s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HADOOP-16677 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985014/HADOOP-16677.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7a4c20f05379 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ee8addb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16633/testReport/ |
| Max. process+thread count | 1345 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16633/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Recalculate the remaining timeout millis correctly while throwing an
> InterupptedException in SocketIOWithTimeout.
>
[jira] [Commented] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968062#comment-16968062
]
Hadoop QA commented on HADOOP-16686:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 30m
7s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
9s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 12 new + 29 unchanged - 57 fixed = 41 total (was 86) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
53s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1696/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/1696 |
| JIRA Issue | HADOOP-16686 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e8f49288be62 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ee8addb |
| Default Java | 1.8.0_222 |
| checkstyle |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1696/2/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1696/2/testReport/ |
| Max. process+thread
[GitHub] [hadoop] hadoop-yetus commented on issue #1696: HADOOP-16686: Review Writable Classes
hadoop-yetus commented on issue #1696: HADOOP-16686: Review Writable Classes URL: https://github.com/apache/hadoop/pull/1696#issuecomment-550131672 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1807 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1091 | trunk passed | | +1 | compile | 1031 | trunk passed | | +1 | checkstyle | 51 | trunk passed | | +1 | mvnsite | 85 | trunk passed | | +1 | shadedclient | 899 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 90 | trunk passed | | 0 | spotbugs | 129 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 127 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 50 | the patch passed | | +1 | compile | 981 | the patch passed | | +1 | javac | 980 | the patch passed | | -0 | checkstyle | 50 | hadoop-common-project/hadoop-common: The patch generated 12 new + 29 unchanged - 57 fixed = 41 total (was 86) | | +1 | mvnsite | 80 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 756 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 89 | the patch passed | | +1 | findbugs | 131 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 556 | hadoop-common in the patch passed. | | +1 | asflicense | 53 | The patch does not generate ASF License warnings. | | | | 8032 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1696/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1696 | | JIRA Issue | HADOOP-16686 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e8f49288be62 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ee8addb | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1696/2/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1696/2/testReport/ | | Max. process+thread count | 1389 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1696/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16677) Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout.
[
https://issues.apache.org/jira/browse/HADOOP-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968053#comment-16968053
]
Wei-Chiu Chuang commented on HADOOP-16677:
--
LGTM Great find. The patch passed precommit test in the PR.
> Recalculate the remaining timeout millis correctly while throwing an
> InterupptedException in SocketIOWithTimeout.
> -
>
> Key: HADOOP-16677
> URL: https://issues.apache.org/jira/browse/HADOOP-16677
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.1.3
>Reporter: Xudong Cao
>Assignee: Xudong Cao
>Priority: Minor
> Labels: pull-request-available
> Attachments: HADOOP-16677.001.patch
>
>
> In SocketIOWithTimeout, when a thread was interrupted and exit from select(),
> it proceed to throw an InterruptedIOException, in exception message the
> remaining timeout mills should be calcuated correctly rather than simply give
> a total timeout millis , which could be very misleading.
>
> An example log before this jira:
> {code:java}
> 2019-10-31 16:20:39,496 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exception for BP-1895911599-192.168.202.11-1572488735889:blk_1073741833_1016
> java.io.InterruptedIOException: Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/192.168.202.12:50010
> remote=/192.168.202.11:57006]. 6 millis timeout left.
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:351)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)){code}
>
> An example log after this jira:
> {code:java}
> 2019-10-31 16:20:39,496 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exception for BP-1895911599-192.168.202.11-1572488735889:blk_1073741833_1016
> java.io.InterruptedIOException: Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/192.168.202.12:50010
> remote=/192.168.202.11:57006]. Total timeout mills is 6, 1000 millis
> timeout left.
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:351)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16676) Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to branch-3.2)
[
https://issues.apache.org/jira/browse/HADOOP-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968049#comment-16968049
]
Hadoop QA commented on HADOOP-16676:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
8s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
36s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
29s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
58s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-client-modules/hadoop-client-minicluster
hadoop-client-modules/hadoop-client-runtime {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
14s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
4s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 12m
47s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-client-modules/hadoop-client-runtime
hadoop-client-modules/hadoop-client-minicluster {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
25s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}111m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
25s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 21s{color}
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green}
[jira] [Commented] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968041#comment-16968041
]
Hadoop QA commented on HADOOP-16686:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 12 new + 29 unchanged - 57 fixed = 41 total (was 86) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
35s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HADOOP-16686 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985012/HADOOP-16686.2.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4cfb9a4c8b49 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ee8addb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16632/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16632/testReport/ |
| Max. process+thread count | 1376 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16632/console |
| Powered by | Apache Yetus
[jira] [Updated] (HADOOP-16677) Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout.
[
https://issues.apache.org/jira/browse/HADOOP-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xudong Cao updated HADOOP-16677:
Description:
In SocketIOWithTimeout, when a thread was interrupted and exit from select(),
it proceed to throw an InterruptedIOException, in exception message the
remaining timeout mills should be calcuated correctly rather than simply give a
total timeout millis , which could be very misleading.
An example log before this jira:
{code:java}
2019-10-31 16:20:39,496 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Exception for BP-1895911599-192.168.202.11-1572488735889:blk_1073741833_1016
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[connected local=/192.168.202.12:50010
remote=/192.168.202.11:57006]. 6 millis timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:351)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)){code}
An example log after this jira:
{code:java}
2019-10-31 16:20:39,496 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Exception for BP-1895911599-192.168.202.11-1572488735889:blk_1073741833_1016
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[connected local=/192.168.202.12:50010
remote=/192.168.202.11:57006]. Total timeout mills is 6, 1000 millis
timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:351)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)){code}
was:
In SocketIOWithTimeout, when a thread was interrupted and exit from select(),
it proceed to throw an InterruptedIOException, in exception message the
remaining timeout mills should be calcuated correctly rather than simply give a
total timeout millis , which could be very misleading.
An example after this jira:
{code:java}
2019-10-31 16:20:39,496 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Exception for BP-1895911599-192.168.202.11-1572488735889:blk_1073741833_1016
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[connected local=/192.168.202.12:50010
remote=/192.168.202.11:57006]. Total timeout mills is 6, 1000 millis
timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:351)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)){code}
> Recalculate the remaining timeout millis correctly while throwing an
> InterupptedException in SocketIOWithTimeout.
> -
>
> Key: HADOOP-16677
> URL: https://issues.apache.org/jira/browse/HADOOP-16677
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.1.3
>Reporter: Xudong Cao
>Assignee: Xudong Cao
>Priority: Minor
> Labels: pull-request-available
> Attachments: HADOOP-16677.001.patch
>
>
> In SocketIOWithTimeout, when a thread was interrupted and exit from select(),
> it proceed to throw an InterruptedIOException, in exception message the
> remaining timeout mills should be calcuated correctly rather than simply give
> a total timeout millis , which could be very misleading.
>
> An example log before this jira:
> {code:java}
> 2019-10-31 16:20:39,496 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exception for BP-1895911599-192.168.202.11-1572488735889:blk_1073741833_1016
> java.io.InterruptedIOException: Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/192.168.202.12:50010
> remote=/192.168.202.11:57006]. 6 millis timeout left.
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:351)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)){code}
>
> An example log after this jira:
> {code:java}
> 2019-10-31 16:20:39,496 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exception for BP-1895911599-192.168.202.11-1572488735889:blk_1073741833_1016
> java.io.InterruptedIOException: Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/192.168.202.12:50010
> remote=/192.168.202.11:57006]. Total timeout mills is 6, 1000 millis
> timeout left.
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:351)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
[jira] [Updated] (HADOOP-16677) Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout.
[
https://issues.apache.org/jira/browse/HADOOP-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xudong Cao updated HADOOP-16677:
Description:
In SocketIOWithTimeout, when a thread was interrupted and exit from select(),
it proceed to throw an InterruptedIOException, in exception message the
remaining timeout mills should be calcuated correctly rather than simply give a
total timeout millis , which could be very misleading.
*An example after this jira:*
{code:java}
2019-10-31 16:20:39,496 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Exception for BP-1895911599-192.168.202.11-1572488735889:blk_1073741833_1016
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[connected local=/192.168.202.12:50010
remote=/192.168.202.11:57006]. Total timeout mills is 6, 1000 millis
timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:351)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)){code}
was:
In SocketIOWithTimeout, when a thread was interrupted and exit from select(),
it proceed to throw an InterruptedIOException, in exception message the
remaining timeout mills should be calcuated correctly rather than simply give a
total timeout millis , which could be very misleading.
An example after this jira:
2019-10-24 09:22:58,212 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Exception for
BP-753871533-10.215.131.216-1511957392115:blk_10646544613_9573675038
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[connected local=/10.196.146.114:9003
remote=/10.215.153.105:38559]. 6 millis timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
...
> Recalculate the remaining timeout millis correctly while throwing an
> InterupptedException in SocketIOWithTimeout.
> -
>
> Key: HADOOP-16677
> URL: https://issues.apache.org/jira/browse/HADOOP-16677
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.1.3
>Reporter: Xudong Cao
>Assignee: Xudong Cao
>Priority: Minor
> Labels: pull-request-available
> Attachments: HADOOP-16677.001.patch
>
>
> In SocketIOWithTimeout, when a thread was interrupted and exit from select(),
> it proceed to throw an InterruptedIOException, in exception message the
> remaining timeout mills should be calcuated correctly rather than simply give
> a total timeout millis , which could be very misleading.
>
> *An example after this jira:*
> {code:java}
> 2019-10-31 16:20:39,496 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exception for BP-1895911599-192.168.202.11-1572488735889:blk_1073741833_1016
> java.io.InterruptedIOException: Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/192.168.202.12:50010
> remote=/192.168.202.11:57006]. Total timeout mills is 6, 1000 millis
> timeout left.
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:351)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16656) Document FairCallQueue configs in core-default.xml
[
https://issues.apache.org/jira/browse/HADOOP-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968034#comment-16968034
]
Hadoop QA commented on HADOOP-16656:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
4s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
23s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
1s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HADOOP-16656 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985006/HADOOP-16656.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux ed2184b723a0 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ee8addb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16631/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16631/testReport/ |
| Max. process+thread count | 1790 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
[jira] [Updated] (HADOOP-16677) Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout.
[
https://issues.apache.org/jira/browse/HADOOP-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xudong Cao updated HADOOP-16677:
Description:
In SocketIOWithTimeout, when a thread was interrupted and exit from select(),
it proceed to throw an InterruptedIOException, in exception message the
remaining timeout mills should be calcuated correctly rather than simply give a
total timeout millis , which could be very misleading.
An example after this jira:
{code:java}
2019-10-31 16:20:39,496 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Exception for BP-1895911599-192.168.202.11-1572488735889:blk_1073741833_1016
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[connected local=/192.168.202.12:50010
remote=/192.168.202.11:57006]. Total timeout mills is 6, 1000 millis
timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:351)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)){code}
was:
In SocketIOWithTimeout, when a thread was interrupted and exit from select(),
it proceed to throw an InterruptedIOException, in exception message the
remaining timeout mills should be calcuated correctly rather than simply give a
total timeout millis , which could be very misleading.
*An example after this jira:*
{code:java}
2019-10-31 16:20:39,496 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Exception for BP-1895911599-192.168.202.11-1572488735889:blk_1073741833_1016
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[connected local=/192.168.202.12:50010
remote=/192.168.202.11:57006]. Total timeout mills is 6, 1000 millis
timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:351)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)){code}
> Recalculate the remaining timeout millis correctly while throwing an
> InterupptedException in SocketIOWithTimeout.
> -
>
> Key: HADOOP-16677
> URL: https://issues.apache.org/jira/browse/HADOOP-16677
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.1.3
>Reporter: Xudong Cao
>Assignee: Xudong Cao
>Priority: Minor
> Labels: pull-request-available
> Attachments: HADOOP-16677.001.patch
>
>
> In SocketIOWithTimeout, when a thread was interrupted and exit from select(),
> it proceed to throw an InterruptedIOException, in exception message the
> remaining timeout mills should be calcuated correctly rather than simply give
> a total timeout millis , which could be very misleading.
>
> An example after this jira:
> {code:java}
> 2019-10-31 16:20:39,496 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exception for BP-1895911599-192.168.202.11-1572488735889:blk_1073741833_1016
> java.io.InterruptedIOException: Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/192.168.202.12:50010
> remote=/192.168.202.11:57006]. Total timeout mills is 6, 1000 millis
> timeout left.
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:351)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16677) Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout.
[ https://issues.apache.org/jira/browse/HADOOP-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HADOOP-16677: Description: In SocketIOWithTimeout, when a thread was interrupted and exit from select(), it proceed to throw an InterruptedIOException, in exception message the remaining timeout mills should be calcuated correctly rather than simply give a total timeout millis , which could be very misleading. An example after this jira: 2019-10-24 09:22:58,212 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-753871533-10.215.131.216-1511957392115:blk_10646544613_9573675038 java.io.InterruptedIOException: Interrupted while waiting for IO on channel java.nio.channels.SocketChannel[connected local=/10.196.146.114:9003 remote=/10.215.153.105:38559]. 6 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) ... was:In SocketIOWithTimeout, when a thread was interrupted and exit from select(), it proceed to throw an InterruptedIOException, in exception message the remaining timeout mills should be calcuated correctly rather than simply give a total timeout millis , which could be very misleading. > Recalculate the remaining timeout millis correctly while throwing an > InterupptedException in SocketIOWithTimeout. > - > > Key: HADOOP-16677 > URL: https://issues.apache.org/jira/browse/HADOOP-16677 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.1.3 >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Minor > Labels: pull-request-available > Attachments: HADOOP-16677.001.patch > > > In SocketIOWithTimeout, when a thread was interrupted and exit from select(), > it proceed to throw an InterruptedIOException, in exception message the > remaining timeout mills should be calcuated correctly rather than simply give > a total timeout millis , which could be very misleading. > > An example after this jira: > 2019-10-24 09:22:58,212 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Exception for > BP-753871533-10.215.131.216-1511957392115:blk_10646544613_9573675038 > java.io.InterruptedIOException: Interrupted while waiting for IO on channel > java.nio.channels.SocketChannel[connected local=/10.196.146.114:9003 > remote=/10.215.153.105:38559]. 6 millis timeout left. > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342) > ... -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16677) Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout.
[ https://issues.apache.org/jira/browse/HADOOP-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xudong Cao updated HADOOP-16677: Attachment: HADOOP-16677.001.patch > Recalculate the remaining timeout millis correctly while throwing an > InterupptedException in SocketIOWithTimeout. > - > > Key: HADOOP-16677 > URL: https://issues.apache.org/jira/browse/HADOOP-16677 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.1.3 >Reporter: Xudong Cao >Assignee: Xudong Cao >Priority: Minor > Labels: pull-request-available > Attachments: HADOOP-16677.001.patch > > > In SocketIOWithTimeout, when a thread was interrupted and exit from select(), > it proceed to throw an InterruptedIOException, in exception message the > remaining timeout mills should be calcuated correctly rather than simply give > a total timeout millis , which could be very misleading. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16685) FileSystem#listStatusIterator does not check if given path exists
[ https://issues.apache.org/jira/browse/HADOOP-16685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968007#comment-16968007 ] Sahil Takiar commented on HADOOP-16685: --- Looks like pre-commit is passing now. The one failing unit test looks unrelated, and is failing in other Hadoop QA runs as well. CC: [~weichiu], [~ste...@apache.org] > FileSystem#listStatusIterator does not check if given path exists > - > > Key: HADOOP-16685 > URL: https://issues.apache.org/jira/browse/HADOOP-16685 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > The Javadocs of FileSystem#listStatusIterator(final Path p) state that it > "@throws FileNotFoundException if p does not exist". However, > that does not seem to be the case. The method simply creates a > DirListingIterator which doesn't do an existence check. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967998#comment-16967998
]
Íñigo Goiri commented on HADOOP-16686:
--
Can we take the chance to fix the checkstyle issues?
BTW, let's either do patches or PRs but not both.
> Review Writable Classes
> ---
>
> Key: HADOOP-16686
> URL: https://issues.apache.org/jira/browse/HADOOP-16686
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HADOOP-16686.1.patch, HADOOP-16686.2.patch
>
>
> Based on my work in HADOOP-16678, I figured I'd look at some of the other
> {{Writeable}} classes. I wanted these looking good because they are so
> core/base to the Hadoop framework.
> Formatted the classes to remove checkstyle violations, and spiffed things up
> where I saw fit. This includes improving some {{equals}} and {{hashcode}}
> implementation. For example:
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/LongWritable.java#L66]
>
> [https://stackoverflow.com/questions/4045063/how-should-i-map-long-to-int-in-hashcode]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967997#comment-16967997
]
David Mollitor edited comment on HADOOP-16686 at 11/6/19 1:18 AM:
--
Including new patch that fixes unit test.
Unit test was testing for specific values returned from the {{compare()}}
method, even though the only guarantees is that the method will return a
negative integer, zero, or a positive integer. One of the changes changed the
value returned, but it was still the correct sign.
https://docs.oracle.com/javase/8/docs/api/java/lang/Comparable.html#compareTo-T-
was (Author: belugabehr):
Including new patch that fixes unit test.
Unit test was testing for specific values returned from the {{compare()}}
method, even though the only guarantees is that the method will return a
negative integer, zero, or a positive integer.
https://docs.oracle.com/javase/8/docs/api/java/lang/Comparable.html#compareTo-T-
> Review Writable Classes
> ---
>
> Key: HADOOP-16686
> URL: https://issues.apache.org/jira/browse/HADOOP-16686
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HADOOP-16686.1.patch, HADOOP-16686.2.patch
>
>
> Based on my work in HADOOP-16678, I figured I'd look at some of the other
> {{Writeable}} classes. I wanted these looking good because they are so
> core/base to the Hadoop framework.
> Formatted the classes to remove checkstyle violations, and spiffed things up
> where I saw fit. This includes improving some {{equals}} and {{hashcode}}
> implementation. For example:
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/LongWritable.java#L66]
>
> [https://stackoverflow.com/questions/4045063/how-should-i-map-long-to-int-in-hashcode]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HADOOP-16686:
Status: Patch Available (was: Open)
Including new patch that fixes unit test.
Unit test was testing for specific values returned from the {{compare()}}
method, even though the only guarantees is that the method will return a
negative integer, zero, or a positive integer.
https://docs.oracle.com/javase/8/docs/api/java/lang/Comparable.html#compareTo-T-
> Review Writable Classes
> ---
>
> Key: HADOOP-16686
> URL: https://issues.apache.org/jira/browse/HADOOP-16686
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HADOOP-16686.1.patch, HADOOP-16686.2.patch
>
>
> Based on my work in HADOOP-16678, I figured I'd look at some of the other
> {{Writeable}} classes. I wanted these looking good because they are so
> core/base to the Hadoop framework.
> Formatted the classes to remove checkstyle violations, and spiffed things up
> where I saw fit. This includes improving some {{equals}} and {{hashcode}}
> implementation. For example:
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/LongWritable.java#L66]
>
> [https://stackoverflow.com/questions/4045063/how-should-i-map-long-to-int-in-hashcode]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HADOOP-16686:
Status: Open (was: Patch Available)
> Review Writable Classes
> ---
>
> Key: HADOOP-16686
> URL: https://issues.apache.org/jira/browse/HADOOP-16686
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HADOOP-16686.1.patch, HADOOP-16686.2.patch
>
>
> Based on my work in HADOOP-16678, I figured I'd look at some of the other
> {{Writeable}} classes. I wanted these looking good because they are so
> core/base to the Hadoop framework.
> Formatted the classes to remove checkstyle violations, and spiffed things up
> where I saw fit. This includes improving some {{equals}} and {{hashcode}}
> implementation. For example:
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/LongWritable.java#L66]
>
> [https://stackoverflow.com/questions/4045063/how-should-i-map-long-to-int-in-hashcode]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HADOOP-16686:
Attachment: HADOOP-16686.2.patch
> Review Writable Classes
> ---
>
> Key: HADOOP-16686
> URL: https://issues.apache.org/jira/browse/HADOOP-16686
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HADOOP-16686.1.patch, HADOOP-16686.2.patch
>
>
> Based on my work in HADOOP-16678, I figured I'd look at some of the other
> {{Writeable}} classes. I wanted these looking good because they are so
> core/base to the Hadoop framework.
> Formatted the classes to remove checkstyle violations, and spiffed things up
> where I saw fit. This includes improving some {{equals}} and {{hashcode}}
> implementation. For example:
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/LongWritable.java#L66]
>
> [https://stackoverflow.com/questions/4045063/how-should-i-map-long-to-int-in-hashcode]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16685) FileSystem#listStatusIterator does not check if given path exists
[
https://issues.apache.org/jira/browse/HADOOP-16685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967983#comment-16967983
]
Hadoop QA commented on HADOOP-16685:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
8s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 13s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
59s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.conf.TestCommonConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/1695 |
| JIRA Issue | HADOOP-16685 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 6aea78f05e35 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ed302f1 |
| Default Java | 1.8.0_222 |
| unit |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/2/testReport/ |
| Max. process+thread count | 1344 (vs. ulimit
[GitHub] [hadoop] hadoop-yetus commented on issue #1695: HADOOP-16685: FileSystem#listStatusIterator does not check if given path exists
hadoop-yetus commented on issue #1695: HADOOP-16685: FileSystem#listStatusIterator does not check if given path exists URL: https://github.com/apache/hadoop/pull/1695#issuecomment-550089399 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 103 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1385 | trunk passed | | +1 | compile | 1156 | trunk passed | | +1 | checkstyle | 55 | trunk passed | | +1 | mvnsite | 100 | trunk passed | | +1 | shadedclient | 994 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 84 | trunk passed | | 0 | spotbugs | 128 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 126 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 50 | the patch passed | | +1 | compile | 1074 | the patch passed | | +1 | javac | 1074 | the patch passed | | +1 | checkstyle | 65 | the patch passed | | +1 | mvnsite | 106 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 925 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 105 | the patch passed | | +1 | findbugs | 163 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 613 | hadoop-common in the patch failed. | | +1 | asflicense | 59 | The patch does not generate ASF License warnings. | | | | 7230 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.conf.TestCommonConfigurationFields | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1695 | | JIRA Issue | HADOOP-16685 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 6aea78f05e35 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ed302f1 | | Default Java | 1.8.0_222 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/2/testReport/ | | Max. process+thread count | 1344 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16656) Document FairCallQueue configs in core-default.xml
[
https://issues.apache.org/jira/browse/HADOOP-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HADOOP-16656:
Attachment: HADOOP-16656.003.patch
Status: Patch Available (was: Reopened)
[~weichiu] Sure. Uploaded patch v003 to ignore FairCallQueue config keys checks
by adding them to {{xmlPropsToSkipCompare}} one by one (since
{{xmlPropsToSkipCompare.add("ipc.[port_number].")}} doesn't work).
> Document FairCallQueue configs in core-default.xml
> --
>
> Key: HADOOP-16656
> URL: https://issues.apache.org/jira/browse/HADOOP-16656
> Project: Hadoop Common
> Issue Type: Task
> Components: conf, documentation
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16656.001.patch, HADOOP-16656.002.patch,
> HADOOP-16656.003.patch
>
>
> So far those callqueue / scheduler / faircallqueue -related configurations
> are only documented in FairCallQueue.md in 3.3.0:
> https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-project-dist/hadoop-common/FairCallQueue.html#Full_List_of_Configurations
> (Thanks Akira for uploading this.)
> Goal: Document those configs in core-default.xml as well to make it easier
> for users(admins) to find and use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16656) Document FairCallQueue configs in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967961#comment-16967961 ] Hudson commented on HADOOP-16656: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17611 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17611/]) Revert "HADOOP-16656. Document FairCallQueue configs in (weichiu: rev ee8addbec45cad7570f73abe97f07943edeeba4d) * (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml > Document FairCallQueue configs in core-default.xml > -- > > Key: HADOOP-16656 > URL: https://issues.apache.org/jira/browse/HADOOP-16656 > Project: Hadoop Common > Issue Type: Task > Components: conf, documentation >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16656.001.patch, HADOOP-16656.002.patch > > > So far those callqueue / scheduler / faircallqueue -related configurations > are only documented in FairCallQueue.md in 3.3.0: > https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-project-dist/hadoop-common/FairCallQueue.html#Full_List_of_Configurations > (Thanks Akira for uploading this.) > Goal: Document those configs in core-default.xml as well to make it easier > for users(admins) to find and use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16656) Document FairCallQueue configs in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HADOOP-16656: Fix Version/s: (was: 3.3.0) > Document FairCallQueue configs in core-default.xml > -- > > Key: HADOOP-16656 > URL: https://issues.apache.org/jira/browse/HADOOP-16656 > Project: Hadoop Common > Issue Type: Task > Components: conf, documentation >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16656.001.patch, HADOOP-16656.002.patch > > > So far those callqueue / scheduler / faircallqueue -related configurations > are only documented in FairCallQueue.md in 3.3.0: > https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-project-dist/hadoop-common/FairCallQueue.html#Full_List_of_Configurations > (Thanks Akira for uploading this.) > Goal: Document those configs in core-default.xml as well to make it easier > for users(admins) to find and use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16656) Document FairCallQueue configs in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967953#comment-16967953 ] Wei-Chiu Chuang commented on HADOOP-16656: -- Reopen the issue. Sorry for my sloppiness. I reverted the commit from trunk. [~smeng] would you please submit a new patch? > Document FairCallQueue configs in core-default.xml > -- > > Key: HADOOP-16656 > URL: https://issues.apache.org/jira/browse/HADOOP-16656 > Project: Hadoop Common > Issue Type: Task > Components: conf, documentation >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16656.001.patch, HADOOP-16656.002.patch > > > So far those callqueue / scheduler / faircallqueue -related configurations > are only documented in FairCallQueue.md in 3.3.0: > https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-project-dist/hadoop-common/FairCallQueue.html#Full_List_of_Configurations > (Thanks Akira for uploading this.) > Goal: Document those configs in core-default.xml as well to make it easier > for users(admins) to find and use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-16656) Document FairCallQueue configs in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HADOOP-16656: -- > Document FairCallQueue configs in core-default.xml > -- > > Key: HADOOP-16656 > URL: https://issues.apache.org/jira/browse/HADOOP-16656 > Project: Hadoop Common > Issue Type: Task > Components: conf, documentation >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16656.001.patch, HADOOP-16656.002.patch > > > So far those callqueue / scheduler / faircallqueue -related configurations > are only documented in FairCallQueue.md in 3.3.0: > https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-project-dist/hadoop-common/FairCallQueue.html#Full_List_of_Configurations > (Thanks Akira for uploading this.) > Goal: Document those configs in core-default.xml as well to make it easier > for users(admins) to find and use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16656) Document FairCallQueue configs in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967952#comment-16967952 ] Siyao Meng commented on HADOOP-16656: - Thanks [~xkrogen] for reporting this. [~weichiu] would revert this from trunk first. I will get a new patch ready. > Document FairCallQueue configs in core-default.xml > -- > > Key: HADOOP-16656 > URL: https://issues.apache.org/jira/browse/HADOOP-16656 > Project: Hadoop Common > Issue Type: Task > Components: conf, documentation >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16656.001.patch, HADOOP-16656.002.patch > > > So far those callqueue / scheduler / faircallqueue -related configurations > are only documented in FairCallQueue.md in 3.3.0: > https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-project-dist/hadoop-common/FairCallQueue.html#Full_List_of_Configurations > (Thanks Akira for uploading this.) > Goal: Document those configs in core-default.xml as well to make it easier > for users(admins) to find and use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967949#comment-16967949
]
Hadoop QA commented on HADOOP-16686:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
54s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 8 new + 19 unchanged - 45 fixed = 27 total (was 64) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 37s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.io.TestWritable |
| | hadoop.conf.TestCommonConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HADOOP-16686 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984993/HADOOP-16686.1.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e2f04c516ff6 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ed302f1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16629/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
[jira] [Commented] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967946#comment-16967946
]
Hadoop QA commented on HADOOP-16686:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 28m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
11s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m
5s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 8 new + 20 unchanged - 45 fixed = 28 total (was 65) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 55s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.io.TestWritable |
| | hadoop.conf.TestCommonConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1696/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/1696 |
| JIRA Issue | HADOOP-16686 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f7af71ad3445 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ed302f1 |
| Default Java | 1.8.0_222 |
| checkstyle |
[GitHub] [hadoop] hadoop-yetus commented on issue #1696: HADOOP-16686: Review Writable Classes
hadoop-yetus commented on issue #1696: HADOOP-16686: Review Writable Classes URL: https://github.com/apache/hadoop/pull/1696#issuecomment-550070384 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1716 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1206 | trunk passed | | +1 | compile | 1056 | trunk passed | | +1 | checkstyle | 46 | trunk passed | | +1 | mvnsite | 80 | trunk passed | | +1 | shadedclient | 961 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 83 | trunk passed | | 0 | spotbugs | 131 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 130 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 54 | the patch passed | | +1 | compile | 1085 | the patch passed | | +1 | javac | 1085 | the patch passed | | -0 | checkstyle | 45 | hadoop-common-project/hadoop-common: The patch generated 8 new + 20 unchanged - 45 fixed = 28 total (was 65) | | +1 | mvnsite | 78 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 825 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 82 | the patch passed | | +1 | findbugs | 132 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 535 | hadoop-common in the patch failed. | | +1 | asflicense | 45 | The patch does not generate ASF License warnings. | | | | 8239 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.io.TestWritable | | | hadoop.conf.TestCommonConfigurationFields | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1696/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1696 | | JIRA Issue | HADOOP-16686 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux f7af71ad3445 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ed302f1 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1696/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1696/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1696/1/testReport/ | | Max. process+thread count | 1454 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1696/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sahilTakiar commented on issue #1695: HADOOP-16685: FileSystem#listStatusIterator does not check if given path exists
sahilTakiar commented on issue #1695: HADOOP-16685: FileSystem#listStatusIterator does not check if given path exists URL: https://github.com/apache/hadoop/pull/1695#issuecomment-550058966 Fixed the checkstyle issue. `TestCommonConfigurationFields` failure seems unrelated and looks like it is failing for other patches as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16676) Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to branch-3.2)
[ https://issues.apache.org/jira/browse/HADOOP-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HADOOP-16676: Attachment: HADOOP-16676.branch-3.2.001.patch Status: Patch Available (was: In Progress) Thanks for pointing that out. Turns out I used "git diff" by mistake (should use "git show") on my local commit. As a result I uploaded the revert of YARN-9949 since I was testing the compilation locally. Now that YARN-9949 is already reverted. It shouldn't be a problem. Uploading the right patch. > Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to > branch-3.2) > - > > Key: HADOOP-16676 > URL: https://issues.apache.org/jira/browse/HADOOP-16676 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.1 >Reporter: DW >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16676.branch-3.2.001.patch, > HADOOP-16676.branch-3.2.001.patch > > > Hello, > > org.apache.hadoop:hadoop-common define the dependency to jetty-webapp and > jetty-xml in version v9.3.24 with known CVE-2017-9735. Please can you upgrade > to version 9.4.7 or higher? > +--- org.apache.hadoop:hadoop-client:3.2.1 > | +--- org.apache.hadoop:hadoop-common:3.2.1 > | +--- org.eclipse.jetty:jetty-webapp:9.3.24.v20180605 > | | | +--- org.eclipse.jetty:jetty-xml:9.3.24.v20180605 > | | | \--- org.eclipse.jetty:jetty-servlet:9.3.24.v20180605 (*) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16676) Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to branch-3.2)
[ https://issues.apache.org/jira/browse/HADOOP-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HADOOP-16676: Status: In Progress (was: Patch Available) > Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to > branch-3.2) > - > > Key: HADOOP-16676 > URL: https://issues.apache.org/jira/browse/HADOOP-16676 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.1 >Reporter: DW >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16676.branch-3.2.001.patch > > > Hello, > > org.apache.hadoop:hadoop-common define the dependency to jetty-webapp and > jetty-xml in version v9.3.24 with known CVE-2017-9735. Please can you upgrade > to version 9.4.7 or higher? > +--- org.apache.hadoop:hadoop-client:3.2.1 > | +--- org.apache.hadoop:hadoop-common:3.2.1 > | +--- org.eclipse.jetty:jetty-webapp:9.3.24.v20180605 > | | | +--- org.eclipse.jetty:jetty-xml:9.3.24.v20180605 > | | | \--- org.eclipse.jetty:jetty-servlet:9.3.24.v20180605 (*) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16656) Document FairCallQueue configs in core-default.xml
[
https://issues.apache.org/jira/browse/HADOOP-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967933#comment-16967933
]
Erik Krogen commented on HADOOP-16656:
--
Hi [~weichiu], [~smeng] -- it looks like this broke
{{hadoop.conf.TestCommonConfigurationFields}}. You can see it failing in the
precommit runs for this JIRA, and I see it now failing on trunk. I checked out
trunk before this commit went in and the test succeeds.
Can you please get this fixed?
> Document FairCallQueue configs in core-default.xml
> --
>
> Key: HADOOP-16656
> URL: https://issues.apache.org/jira/browse/HADOOP-16656
> Project: Hadoop Common
> Issue Type: Task
> Components: conf, documentation
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16656.001.patch, HADOOP-16656.002.patch
>
>
> So far those callqueue / scheduler / faircallqueue -related configurations
> are only documented in FairCallQueue.md in 3.3.0:
> https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-project-dist/hadoop-common/FairCallQueue.html#Full_List_of_Configurations
> (Thanks Akira for uploading this.)
> Goal: Document those configs in core-default.xml as well to make it easier
> for users(admins) to find and use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on a change in pull request #1694: HADOOP-13327 Output Stream Specification.
hadoop-yetus commented on a change in pull request #1694: HADOOP-13327 Output Stream Specification. URL: https://github.com/apache/hadoop/pull/1694#discussion_r342817373 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,889 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + +## Introduction + +This document covers the Output Streams within the context of the +[Hadoop File System Specification](index.html). + +It uses the filesystem model defined in [A Model of a Hadoop Filesystem](model.html) +with the notation defined in [notation](Notation.md). + +The target audiences are +1. Users of the APIs. While `java.io.OutputStream` is a standard interfaces, +this document clarifies how it is implemented in HDFS and elsewhere. +The Hadoop-specific interfaces `Syncable` and `StreamCapabilities` are new; +`Syncable` is notable in offering durability and visibility guarantees which +exceed that of `OutputStream`. +1. Implementors of File Systems and clients. + +## How data is written to a filesystem + +The core mechanism to write data to files through the Hadoop FileSystem APIs +is through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's `OutputStream` implementations +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported its methods. However, this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS do not support the Sync operations, +even though they are implemented as subclass of an output stream which is `Syncable`. + +A new interface: `StreamCapabilities`. This allows callers +to probe the exact capabilities of a stream, even transitively +through a chain of streams. + + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte] +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path`, can be tracked to form a triple +`path, open, buffer` + +```python +Stream = (path: Path, open: Boolean, buffer: byte[]) +``` + + Visibility of Flushed Data + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path MUST match that of +`buffer`. That is, the following condition MUST hold: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path MUST see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability +guarantees, not visibility of data. + +### State of Stream and Filesystem after `Filesystem.create()` + +The output stream returned by a `FileSystem.create(path)` or +`FileSystem.createFile(path).build` +can be modeled as a triple containing an empty array of no data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` MUST contain a 0-byte file at the path: + +```python +data(FS', path) == [] +``` + +Thus, the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: see caveats in the "Object Stores" section below. + +### State of Stream and Filesystem after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in the `close()` operation, or at any other +time the stream chooses to do so, the contents of the file +are replaced with the current buffer + +```python +Stream' = (path, true, buffer) +FS' = FS where data(FS', path) == buffer +``` + +After a call to `close()`, the stream is closed for all operations other +than `close()`; they MAY fail with `IOException` or `RuntimeException`. + +```python +Stream' = (path, false, []) +``` +
[GitHub] [hadoop] hadoop-yetus commented on issue #1694: HADOOP-13327 Output Stream Specification.
hadoop-yetus commented on issue #1694: HADOOP-13327 Output Stream Specification. URL: https://github.com/apache/hadoop/pull/1694#issuecomment-550039278 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 83 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for branch | | +1 | mvninstall | 1210 | trunk passed | | +1 | compile | 1082 | trunk passed | | +1 | checkstyle | 165 | trunk passed | | +1 | mvnsite | 259 | trunk passed | | +1 | shadedclient | 1248 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 258 | trunk passed | | 0 | spotbugs | 50 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 433 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 28 | Maven dependency ordering for patch | | +1 | mvninstall | 174 | the patch passed | | +1 | compile | 1053 | the patch passed | | -1 | javac | 1053 | root generated 2 new + 1892 unchanged - 0 fixed = 1894 total (was 1892) | | -0 | checkstyle | 164 | root: The patch generated 1 new + 77 unchanged - 3 fixed = 78 total (was 80) | | +1 | mvnsite | 258 | the patch passed | | -1 | whitespace | 0 | The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 | xml | 3 | The patch has no ill-formed XML file. | | +1 | shadedclient | 893 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 277 | the patch passed | | +1 | findbugs | 508 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 569 | hadoop-common in the patch failed. | | -1 | unit | 7719 | hadoop-hdfs in the patch failed. | | +1 | unit | 106 | hadoop-azure in the patch passed. | | +1 | unit | 77 | hadoop-azure-datalake in the patch passed. | | +1 | asflicense | 63 | The patch does not generate ASF License warnings. | | | | 16557 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.util.TestReadWriteDiskValidator | | | hadoop.conf.TestCommonConfigurationFields | | | hadoop.hdfs.server.namenode.TestFSImage | | | hadoop.hdfs.server.balancer.TestBalancerRPCDelay | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1694/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1694 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux c5adbb815350 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / bfb8f28 | | Default Java | 1.8.0_222 | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1694/2/artifact/out/diff-compile-javac-root.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1694/2/artifact/out/diff-checkstyle-root.txt | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-1694/2/artifact/out/whitespace-eol.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1694/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1694/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1694/2/testReport/ | | Max. process+thread count | 3874 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-azure hadoop-tools/hadoop-azure-datalake U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1694/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hadoop] hadoop-yetus commented on a change in pull request #1694: HADOOP-13327 Output Stream Specification.
hadoop-yetus commented on a change in pull request #1694: HADOOP-13327 Output Stream Specification. URL: https://github.com/apache/hadoop/pull/1694#discussion_r342817360 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,889 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + +## Introduction + +This document covers the Output Streams within the context of the +[Hadoop File System Specification](index.html). + +It uses the filesystem model defined in [A Model of a Hadoop Filesystem](model.html) +with the notation defined in [notation](Notation.md). + +The target audiences are +1. Users of the APIs. While `java.io.OutputStream` is a standard interfaces, +this document clarifies how it is implemented in HDFS and elsewhere. +The Hadoop-specific interfaces `Syncable` and `StreamCapabilities` are new; +`Syncable` is notable in offering durability and visibility guarantees which +exceed that of `OutputStream`. +1. Implementors of File Systems and clients. + +## How data is written to a filesystem + +The core mechanism to write data to files through the Hadoop FileSystem APIs +is through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's `OutputStream` implementations +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported its methods. However, this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS do not support the Sync operations, +even though they are implemented as subclass of an output stream which is `Syncable`. + +A new interface: `StreamCapabilities`. This allows callers +to probe the exact capabilities of a stream, even transitively +through a chain of streams. + + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte] +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path`, can be tracked to form a triple +`path, open, buffer` + +```python +Stream = (path: Path, open: Boolean, buffer: byte[]) +``` + + Visibility of Flushed Data + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path MUST match that of +`buffer`. That is, the following condition MUST hold: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path MUST see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability +guarantees, not visibility of data. + +### State of Stream and Filesystem after `Filesystem.create()` + +The output stream returned by a `FileSystem.create(path)` or +`FileSystem.createFile(path).build` +can be modeled as a triple containing an empty array of no data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` MUST contain a 0-byte file at the path: + +```python +data(FS', path) == [] +``` + +Thus, the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: see caveats in the "Object Stores" section below. + +### State of Stream and Filesystem after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in the `close()` operation, or at any other +time the stream chooses to do so, the contents of the file +are replaced with the current buffer + +```python +Stream' = (path, true, buffer) +FS' = FS where data(FS', path) == buffer +``` + +After a call to `close()`, the stream is closed for all operations other +than `close()`; they MAY fail with `IOException` or `RuntimeException`. + +```python +Stream' = (path, false, []) +``` +
[GitHub] [hadoop] hadoop-yetus commented on a change in pull request #1694: HADOOP-13327 Output Stream Specification.
hadoop-yetus commented on a change in pull request #1694: HADOOP-13327 Output Stream Specification. URL: https://github.com/apache/hadoop/pull/1694#discussion_r342817346 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,889 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + +## Introduction + +This document covers the Output Streams within the context of the +[Hadoop File System Specification](index.html). + +It uses the filesystem model defined in [A Model of a Hadoop Filesystem](model.html) +with the notation defined in [notation](Notation.md). + +The target audiences are +1. Users of the APIs. While `java.io.OutputStream` is a standard interfaces, +this document clarifies how it is implemented in HDFS and elsewhere. +The Hadoop-specific interfaces `Syncable` and `StreamCapabilities` are new; +`Syncable` is notable in offering durability and visibility guarantees which +exceed that of `OutputStream`. +1. Implementors of File Systems and clients. + +## How data is written to a filesystem + +The core mechanism to write data to files through the Hadoop FileSystem APIs +is through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's `OutputStream` implementations +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported its methods. However, this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS do not support the Sync operations, +even though they are implemented as subclass of an output stream which is `Syncable`. + +A new interface: `StreamCapabilities`. This allows callers +to probe the exact capabilities of a stream, even transitively +through a chain of streams. + + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte] +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path`, can be tracked to form a triple +`path, open, buffer` + +```python +Stream = (path: Path, open: Boolean, buffer: byte[]) +``` + + Visibility of Flushed Data + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path MUST match that of +`buffer`. That is, the following condition MUST hold: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path MUST see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability +guarantees, not visibility of data. + +### State of Stream and Filesystem after `Filesystem.create()` + +The output stream returned by a `FileSystem.create(path)` or +`FileSystem.createFile(path).build` +can be modeled as a triple containing an empty array of no data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` MUST contain a 0-byte file at the path: + +```python +data(FS', path) == [] +``` + +Thus, the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: see caveats in the "Object Stores" section below. + +### State of Stream and Filesystem after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in the `close()` operation, or at any other +time the stream chooses to do so, the contents of the file +are replaced with the current buffer + +```python +Stream' = (path, true, buffer) +FS' = FS where data(FS', path) == buffer +``` + +After a call to `close()`, the stream is closed for all operations other +than `close()`; they MAY fail with `IOException` or `RuntimeException`. + +```python +Stream' = (path, false, []) +``` +
[GitHub] [hadoop] hadoop-yetus commented on a change in pull request #1694: HADOOP-13327 Output Stream Specification.
hadoop-yetus commented on a change in pull request #1694: HADOOP-13327 Output Stream Specification. URL: https://github.com/apache/hadoop/pull/1694#discussion_r342817321 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,889 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + +## Introduction + +This document covers the Output Streams within the context of the +[Hadoop File System Specification](index.html). + +It uses the filesystem model defined in [A Model of a Hadoop Filesystem](model.html) +with the notation defined in [notation](Notation.md). + +The target audiences are +1. Users of the APIs. While `java.io.OutputStream` is a standard interfaces, +this document clarifies how it is implemented in HDFS and elsewhere. +The Hadoop-specific interfaces `Syncable` and `StreamCapabilities` are new; +`Syncable` is notable in offering durability and visibility guarantees which +exceed that of `OutputStream`. +1. Implementors of File Systems and clients. + +## How data is written to a filesystem + +The core mechanism to write data to files through the Hadoop FileSystem APIs +is through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's `OutputStream` implementations +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported its methods. However, this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS do not support the Sync operations, +even though they are implemented as subclass of an output stream which is `Syncable`. + +A new interface: `StreamCapabilities`. This allows callers +to probe the exact capabilities of a stream, even transitively +through a chain of streams. + + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte] +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path`, can be tracked to form a triple +`path, open, buffer` + +```python +Stream = (path: Path, open: Boolean, buffer: byte[]) +``` + + Visibility of Flushed Data + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path MUST match that of +`buffer`. That is, the following condition MUST hold: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path MUST see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability +guarantees, not visibility of data. + +### State of Stream and Filesystem after `Filesystem.create()` + +The output stream returned by a `FileSystem.create(path)` or +`FileSystem.createFile(path).build` +can be modeled as a triple containing an empty array of no data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` MUST contain a 0-byte file at the path: + +```python +data(FS', path) == [] +``` + +Thus, the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: see caveats in the "Object Stores" section below. + +### State of Stream and Filesystem after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in the `close()` operation, or at any other +time the stream chooses to do so, the contents of the file +are replaced with the current buffer + +```python +Stream' = (path, true, buffer) +FS' = FS where data(FS', path) == buffer +``` + +After a call to `close()`, the stream is closed for all operations other +than `close()`; they MAY fail with `IOException` or `RuntimeException`. + +```python +Stream' = (path, false, []) +``` +
[jira] [Updated] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HADOOP-16686:
-
Status: Patch Available (was: Open)
> Review Writable Classes
> ---
>
> Key: HADOOP-16686
> URL: https://issues.apache.org/jira/browse/HADOOP-16686
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HADOOP-16686.1.patch
>
>
> Based on my work in HADOOP-16678, I figured I'd look at some of the other
> {{Writeable}} classes. I wanted these looking good because they are so
> core/base to the Hadoop framework.
> Formatted the classes to remove checkstyle violations, and spiffed things up
> where I saw fit. This includes improving some {{equals}} and {{hashcode}}
> implementation. For example:
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/LongWritable.java#L66]
>
> [https://stackoverflow.com/questions/4045063/how-should-i-map-long-to-int-in-hashcode]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16685) FileSystem#listStatusIterator does not check if given path exists
[
https://issues.apache.org/jira/browse/HADOOP-16685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967894#comment-16967894
]
Hadoop QA commented on HADOOP-16685:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
8s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
6s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 75 unchanged - 0 fixed = 76 total (was 75) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 15s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}110m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.conf.TestCommonConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/1695 |
| JIRA Issue | HADOOP-16685 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 63c4967f93b8 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / bfb8f28 |
| Default Java | 1.8.0_222 |
| checkstyle |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
[GitHub] [hadoop] hadoop-yetus commented on issue #1695: HADOOP-16685: FileSystem#listStatusIterator does not check if given path exists
hadoop-yetus commented on issue #1695: HADOOP-16685: FileSystem#listStatusIterator does not check if given path exists URL: https://github.com/apache/hadoop/pull/1695#issuecomment-550034685 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 74 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1258 | trunk passed | | +1 | compile | 1065 | trunk passed | | +1 | checkstyle | 47 | trunk passed | | +1 | mvnsite | 81 | trunk passed | | +1 | shadedclient | 955 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 87 | trunk passed | | 0 | spotbugs | 128 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 126 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 50 | the patch passed | | +1 | compile | 1026 | the patch passed | | +1 | javac | 1026 | the patch passed | | -0 | checkstyle | 45 | hadoop-common-project/hadoop-common: The patch generated 1 new + 75 unchanged - 0 fixed = 76 total (was 75) | | +1 | mvnsite | 80 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 816 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 85 | the patch passed | | +1 | findbugs | 135 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 555 | hadoop-common in the patch failed. | | +1 | asflicense | 45 | The patch does not generate ASF License warnings. | | | | 6606 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.conf.TestCommonConfigurationFields | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1695 | | JIRA Issue | HADOOP-16685 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 63c4967f93b8 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / bfb8f28 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/1/testReport/ | | Max. process+thread count | 1346 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1695/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] belugabehr opened a new pull request #1696: HADOOP-16686: Review Writable Classes
belugabehr opened a new pull request #1696: HADOOP-16686: Review Writable Classes URL: https://github.com/apache/hadoop/pull/1696 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HADOOP-16686:
Status: Open (was: Patch Available)
> Review Writable Classes
> ---
>
> Key: HADOOP-16686
> URL: https://issues.apache.org/jira/browse/HADOOP-16686
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HADOOP-16686.1.patch
>
>
> Based on my work in HADOOP-16678, I figured I'd look at some of the other
> {{Writeable}} classes. I wanted these looking good because they are so
> core/base to the Hadoop framework.
> Formatted the classes to remove checkstyle violations, and spiffed things up
> where I saw fit. This includes improving some {{equals}} and {{hashcode}}
> implementation. For example:
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/LongWritable.java#L66]
>
> [https://stackoverflow.com/questions/4045063/how-should-i-map-long-to-int-in-hashcode]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HADOOP-16686:
Status: Patch Available (was: Open)
> Review Writable Classes
> ---
>
> Key: HADOOP-16686
> URL: https://issues.apache.org/jira/browse/HADOOP-16686
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HADOOP-16686.1.patch
>
>
> Based on my work in HADOOP-16678, I figured I'd look at some of the other
> {{Writeable}} classes. I wanted these looking good because they are so
> core/base to the Hadoop framework.
> Formatted the classes to remove checkstyle violations, and spiffed things up
> where I saw fit. This includes improving some {{equals}} and {{hashcode}}
> implementation. For example:
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/LongWritable.java#L66]
>
> [https://stackoverflow.com/questions/4045063/how-should-i-map-long-to-int-in-hashcode]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HADOOP-16686:
Description:
Based on my work in HADOOP-16678, I figured I'd look at some of the other
{{Writeable}} classes. I wanted these looking good because they are so
core/base to the Hadoop framework.
Formatted the classes to remove checkstyle violations, and spiffed things up
where I saw fit. This includes improving some {{equals}} and {{hashcode}}
implementation. For example:
[https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/LongWritable.java#L66]
[https://stackoverflow.com/questions/4045063/how-should-i-map-long-to-int-in-hashcode]
was:
Based on my work in [HADOOP-16678], I figured I'd look at some of the other
{{Writeable}} classes.
Formatted the classes to remove checkstyle violations, and spiffed things up
where I saw fit. This includes improving some {{equals}} and {{hashcode}}
implementation. For example:
[https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/LongWritable.java#L66]
[https://stackoverflow.com/questions/4045063/how-should-i-map-long-to-int-in-hashcode]
> Review Writable Classes
> ---
>
> Key: HADOOP-16686
> URL: https://issues.apache.org/jira/browse/HADOOP-16686
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HADOOP-16686.1.patch
>
>
> Based on my work in HADOOP-16678, I figured I'd look at some of the other
> {{Writeable}} classes. I wanted these looking good because they are so
> core/base to the Hadoop framework.
> Formatted the classes to remove checkstyle violations, and spiffed things up
> where I saw fit. This includes improving some {{equals}} and {{hashcode}}
> implementation. For example:
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/LongWritable.java#L66]
>
> [https://stackoverflow.com/questions/4045063/how-should-i-map-long-to-int-in-hashcode]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16686) Review Writable Classes
David Mollitor created HADOOP-16686:
---
Summary: Review Writable Classes
Key: HADOOP-16686
URL: https://issues.apache.org/jira/browse/HADOOP-16686
Project: Hadoop Common
Issue Type: Improvement
Components: common
Affects Versions: 3.2
Reporter: David Mollitor
Assignee: David Mollitor
Attachments: HADOOP-16686.1.patch
Based on my work in [HADOOP-16678], I figured I'd look at some of the other
{{Writeable}} classes.
Formatted the classes to remove checkstyle violations, and spiffed things up
where I saw fit. This includes improving some {{equals}} and {{hashcode}}
implementation. For example:
[https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/LongWritable.java#L66]
[https://stackoverflow.com/questions/4045063/how-should-i-map-long-to-int-in-hashcode]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16686) Review Writable Classes
[
https://issues.apache.org/jira/browse/HADOOP-16686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HADOOP-16686:
Attachment: HADOOP-16686.1.patch
> Review Writable Classes
> ---
>
> Key: HADOOP-16686
> URL: https://issues.apache.org/jira/browse/HADOOP-16686
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HADOOP-16686.1.patch
>
>
> Based on my work in [HADOOP-16678], I figured I'd look at some of the other
> {{Writeable}} classes.
> Formatted the classes to remove checkstyle violations, and spiffed things up
> where I saw fit. This includes improving some {{equals}} and {{hashcode}}
> implementation. For example:
>
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/LongWritable.java#L66]
>
> [https://stackoverflow.com/questions/4045063/how-should-i-map-long-to-int-in-hashcode]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sahilTakiar opened a new pull request #1695: HADOOP-16685: FileSystem#listStatusIterator does not check if given path exists
sahilTakiar opened a new pull request #1695: HADOOP-16685: FileSystem#listStatusIterator does not check if given path exists URL: https://github.com/apache/hadoop/pull/1695 [HADOOP-16685](https://issues.apache.org/jira/browse/HADOOP-16685): FileSystem#listStatusIterator does not check if given path exists Changed `DirListingIterator` so that the constructor makes the first call to `listStatusBatch`. This is similar to what `FileSystem#listFiles` does. Added a contract test in `AbstractContractGetFileStatusTest.java` to validate that `listStatusIterator` throws a FNF exception if the given `Path` does not exist. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16676) Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to branch-3.2)
[
https://issues.apache.org/jira/browse/HADOOP-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967817#comment-16967817
]
Hadoop QA commented on HADOOP-16676:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 10m
36s{color} | {color:red} root in branch-3.2 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
21s{color} | {color:red} hadoop-yarn-server-resourcemanager in branch-3.2
failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} branch-3.2 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-server-resourcemanager in branch-3.2
failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 4m
22s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} hadoop-yarn-server-resourcemanager in branch-3.2
failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 35s{color}
| {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 15 new + 2 unchanged - 10 fixed = 17 total (was 12) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 214 unchanged - 7 fixed = 214 total (was 221)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 0 new + 4 unchanged - 2 fixed = 4 total (was 6) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 58s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher |
| |
hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisherForV2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:63396beab41 |
| JIRA Issue | HADOOP-16676 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984982/HADOOP-16676.branch-3.2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8a3cfed8aea4 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HADOOP-16685) FileSystem#listStatusIterator does not check if given path exists
[
https://issues.apache.org/jira/browse/HADOOP-16685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967816#comment-16967816
]
Sahil Takiar commented on HADOOP-16685:
---
{quote}What is the expectation? a fast failure?
{quote}
Yup. Expectation is that calling {{FileSystem#listStatusIterator(final Path
p)}} on a {{Path}} that does not exist should throw a
{{FileNotFoundException}}, which is consistent with what the current javadocs
say + what {{listFiles}} does as well. Currently, {{listStatusIterator}} fails
on the first call to {{hasNext()}} if the specified {{Path}} does not exist.
{quote}does this break with other file systems other than ABFS?
{quote}
Technically it affects all filesystems, except maybe those that have a custom
implementation of {{FileSystem#listStatusIterator(final Path p)}}. It only
happens to affect Impala-on-ABFS, because Impala uses different APIs for
listing out directories depending on the target filesystem - e.g. for S3 it
uses {{listFiles(recursive)}} because S3A has a more performant way to do
recursive listings.
> FileSystem#listStatusIterator does not check if given path exists
> -
>
> Key: HADOOP-16685
> URL: https://issues.apache.org/jira/browse/HADOOP-16685
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> The Javadocs of FileSystem#listStatusIterator(final Path p) state that it
> "@throws FileNotFoundException if p does not exist". However,
> that does not seem to be the case. The method simply creates a
> DirListingIterator which doesn't do an existence check.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16685) FileSystem#listStatusIterator does not check if given path exists
[ https://issues.apache.org/jira/browse/HADOOP-16685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967798#comment-16967798 ] Wei-Chiu Chuang commented on HADOOP-16685: -- does this break with other file systems other than ABFS? > FileSystem#listStatusIterator does not check if given path exists > - > > Key: HADOOP-16685 > URL: https://issues.apache.org/jira/browse/HADOOP-16685 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > The Javadocs of FileSystem#listStatusIterator(final Path p) state that it > "@throws FileNotFoundException if p does not exist". However, > that does not seem to be the case. The method simply creates a > DirListingIterator which doesn't do an existence check. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16681) mvn javadoc:javadoc fails in hadoop-aws
[
https://issues.apache.org/jira/browse/HADOOP-16681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967787#comment-16967787
]
Steve Loughran commented on HADOOP-16681:
-
thanks -sorry for this!
> mvn javadoc:javadoc fails in hadoop-aws
> ---
>
> Key: HADOOP-16681
> URL: https://issues.apache.org/jira/browse/HADOOP-16681
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16681.001.patch
>
>
> mvn javadoc:javadoc fails in hadoop-aws module.
> {code}
> [ERROR]
> /Users/sri/projects/hadoop-mirror/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/NetworkBinding.java:117:
> error: semicolon missing
> [ERROR]*
> https://forums.aws.amazon.com/thread.jspa?messageID=796829=0
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16685) FileSystem#listStatusIterator does not check if given path exists
[ https://issues.apache.org/jira/browse/HADOOP-16685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967786#comment-16967786 ] Steve Loughran commented on HADOOP-16685: - Not look at that operation in any detail. What is the expectation? a fast failure? > FileSystem#listStatusIterator does not check if given path exists > - > > Key: HADOOP-16685 > URL: https://issues.apache.org/jira/browse/HADOOP-16685 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > The Javadocs of FileSystem#listStatusIterator(final Path p) state that it > "@throws FileNotFoundException if p does not exist". However, > that does not seem to be the case. The method simply creates a > DirListingIterator which doesn't do an existence check. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16685) FileSystem#listStatusIterator does not check if given path exists
Sahil Takiar created HADOOP-16685: - Summary: FileSystem#listStatusIterator does not check if given path exists Key: HADOOP-16685 URL: https://issues.apache.org/jira/browse/HADOOP-16685 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Sahil Takiar Assignee: Sahil Takiar The Javadocs of FileSystem#listStatusIterator(final Path p) state that it "@throws FileNotFoundException if p does not exist". However, that does not seem to be the case. The method simply creates a DirListingIterator which doesn't do an existence check. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16676) Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to branch-3.2)
[ https://issues.apache.org/jira/browse/HADOOP-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967753#comment-16967753 ] Wei-Chiu Chuang commented on HADOOP-16676: -- Hey [~smeng] I think you attached the wrong patch. Could you check again? Thanks. > Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to > branch-3.2) > - > > Key: HADOOP-16676 > URL: https://issues.apache.org/jira/browse/HADOOP-16676 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.1 >Reporter: DW >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16676.branch-3.2.001.patch > > > Hello, > > org.apache.hadoop:hadoop-common define the dependency to jetty-webapp and > jetty-xml in version v9.3.24 with known CVE-2017-9735. Please can you upgrade > to version 9.4.7 or higher? > +--- org.apache.hadoop:hadoop-client:3.2.1 > | +--- org.apache.hadoop:hadoop-common:3.2.1 > | +--- org.eclipse.jetty:jetty-webapp:9.3.24.v20180605 > | | | +--- org.eclipse.jetty:jetty-xml:9.3.24.v20180605 > | | | \--- org.eclipse.jetty:jetty-servlet:9.3.24.v20180605 (*) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16684) s3guard bucket info to list a bit more about authoritative paths
Steve Loughran created HADOOP-16684: --- Summary: s3guard bucket info to list a bit more about authoritative paths Key: HADOOP-16684 URL: https://issues.apache.org/jira/browse/HADOOP-16684 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.3.0 Reporter: Steve Loughran Assignee: Steve Loughran Gave a bit more detail about authoritative paths (i.e. resolve them and then print them), and the TTL for authoritative directory markers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification
hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#issuecomment-548927380 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1465 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 6 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 73 | Maven dependency ordering for branch | | +1 | mvninstall | 1232 | trunk passed | | +1 | compile | 1070 | trunk passed | | +1 | checkstyle | 168 | trunk passed | | +1 | mvnsite | 275 | trunk passed | | +1 | shadedclient | 1318 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 265 | trunk passed | | 0 | spotbugs | 43 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 464 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for patch | | +1 | mvninstall | 208 | the patch passed | | +1 | compile | 1129 | the patch passed | | -1 | javac | 1129 | root generated 2 new + 1890 unchanged - 0 fixed = 1892 total (was 1890) | | -0 | checkstyle | 172 | root: The patch generated 6 new + 78 unchanged - 3 fixed = 84 total (was 81) | | +1 | mvnsite | 282 | the patch passed | | -1 | whitespace | 1 | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 | xml | 3 | The patch has no ill-formed XML file. | | +1 | shadedclient | 826 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 257 | the patch passed | | +1 | findbugs | 533 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 603 | hadoop-common in the patch failed. | | -1 | unit | 7305 | hadoop-hdfs in the patch failed. | | +1 | unit | 126 | hadoop-aws in the patch passed. | | +1 | unit | 105 | hadoop-azure in the patch passed. | | +1 | unit | 80 | hadoop-azure-datalake in the patch passed. | | +1 | asflicense | 65 | The patch does not generate ASF License warnings. | | | | 17936 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.conf.TestCommonConfigurationFields | | | hadoop.hdfs.server.balancer.TestBalancerRPCDelay | | | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-575/13/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/575 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 0d94e2762c82 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ef9d12d | | Default Java | 1.8.0_222 | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/13/artifact/out/diff-compile-javac-root.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/13/artifact/out/diff-checkstyle-root.txt | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/13/artifact/out/whitespace-eol.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/13/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/13/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/13/testReport/ | | Max. process+thread count | 3318 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure hadoop-tools/hadoop-azure-datalake U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/13/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967742#comment-16967742
]
Hadoop QA commented on HADOOP-16683:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 53s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}100m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.conf.TestCommonConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HADOOP-16683 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984972/HADOOP-16683.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5779140c0616 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d17ba85 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16627/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16627/testReport/ |
| Max. process+thread count | 1346 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16627/console |
| Powered by | Apache Yetus 0.8.0
[GitHub] [hadoop] lukmajercak commented on issue #1690: HADOOP-16680. Add MicrosoftGraphGroupsMapping GroupMappingServiceProvider
lukmajercak commented on issue #1690: HADOOP-16680. Add MicrosoftGraphGroupsMapping GroupMappingServiceProvider URL: https://github.com/apache/hadoop/pull/1690#issuecomment-549947479 Doesn't spark use hadoop-common for authz though? So if we wanna use msgraph for authz in spark we'll have to add hadoop-azure as a dependency? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16333) Test Hang in S3A S3guard test MetadataStoreTestBase.testListChildren
[
https://issues.apache.org/jira/browse/HADOOP-16333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16333.
-
Resolution: Cannot Reproduce
> Test Hang in S3A S3guard test MetadataStoreTestBase.testListChildren
>
>
> Key: HADOOP-16333
> URL: https://issues.apache.org/jira/browse/HADOOP-16333
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Major
>
> Test hangs to point of timeout in {{MetadataStoreTestBase.testListChildren}}
> on trunk
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification
hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#issuecomment-527286536 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 10 | https://github.com/apache/hadoop/pull/575 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/575 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/12/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification
hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#issuecomment-523064173 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 10 | https://github.com/apache/hadoop/pull/575 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/575 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/9/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification
hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#issuecomment-523935313 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 13 | https://github.com/apache/hadoop/pull/575 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/575 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/10/console | | versions | git=2.7.4 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification
hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#issuecomment-525256724 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 10 | https://github.com/apache/hadoop/pull/575 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/575 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/11/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification
hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#issuecomment-522098204 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 12 | https://github.com/apache/hadoop/pull/575 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/575 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/8/console | | versions | git=2.7.4 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification
hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#issuecomment-513323329 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 75 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 6 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for branch | | +1 | mvninstall | 1095 | trunk passed | | +1 | compile | 1181 | trunk passed | | +1 | checkstyle | 140 | trunk passed | | +1 | mvnsite | 257 | trunk passed | | +1 | shadedclient | 1068 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 193 | trunk passed | | 0 | spotbugs | 41 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 435 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 23 | Maven dependency ordering for patch | | +1 | mvninstall | 188 | the patch passed | | +1 | compile | 1021 | the patch passed | | -1 | javac | 1021 | root generated 2 new + 1477 unchanged - 0 fixed = 1479 total (was 1477) | | -0 | checkstyle | 146 | root: The patch generated 4 new + 78 unchanged - 3 fixed = 82 total (was 81) | | +1 | mvnsite | 280 | the patch passed | | -1 | whitespace | 0 | The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 | xml | 3 | The patch has no ill-formed XML file. | | +1 | shadedclient | 675 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 215 | the patch passed | | -1 | findbugs | 29 | hadoop-aws in the patch failed. | | -1 | findbugs | 27 | hadoop-azure in the patch failed. | | -1 | findbugs | 26 | hadoop-azure-datalake in the patch failed. | ||| _ Other Tests _ | | +1 | unit | 510 | hadoop-common in the patch passed. | | -1 | unit | 6478 | hadoop-hdfs in the patch failed. | | -1 | unit | 32 | hadoop-aws in the patch failed. | | -1 | unit | 33 | hadoop-azure in the patch failed. | | -1 | unit | 31 | hadoop-azure-datalake in the patch failed. | | -1 | asflicense | 47 | The patch generated 1 ASF License warnings. | | | | 14416 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.hdfs.server.datanode.TestDirectoryScanner | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-575/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/575 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 7b3a27325fdb 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / cd967c7 | | Default Java | 1.8.0_212 | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/4/artifact/out/diff-compile-javac-root.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/4/artifact/out/diff-checkstyle-root.txt | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/4/artifact/out/whitespace-eol.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/4/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/4/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/4/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure-datalake.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/4/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/4/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/4/artifact/out/patch-unit-hadoop-tools_hadoop-azure-datalake.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/4/testReport/ | | asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/4/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 4670 (vs.
[jira] [Resolved] (HADOOP-16347) MapReduce job tasks fails on S3A ssl3_get_server_certificate:certificate verify
[
https://issues.apache.org/jira/browse/HADOOP-16347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16347.
-
Resolution: Cannot Reproduce
now that HADOOP-16050 is rolled back, assuming this is fixed. closing
> MapReduce job tasks fails on S3A ssl3_get_server_certificate:certificate
> verify
> ---
>
> Key: HADOOP-16347
> URL: https://issues.apache.org/jira/browse/HADOOP-16347
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Steve Loughran
>Priority: Major
>
> MapReduce Job tasks fails. There are few tasks which fails with below
> exception and few hangs and then times out. List Files on S3 works fine from
> hadoop client.
>
> *Exception from failed task:*
>
> {code}
> 2019-05-30 20:23:05,424 ERROR [IPC Server handler 19 on 35791]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1559246386193_0001_m_00_0 - exited :
> org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on
> qe-cloudstorage-bucket: com.amazonaws.SdkClientException: Unable to execute
> HTTP request: error:14090086:SSL
> routines:ssl3_get_server_certificate:certificate verify failed: Unable to
> execute HTTP request: error:14090086:SSL
> routines:ssl3_get_server_certificate:certificate verify failed
> at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:204)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:314)
> at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:406)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:310)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:285)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:444)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:350)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3315)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3364)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3332)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:491)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:161)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:117)
> at
> org.apache.hadoop.examples.terasort.TeraOutputFormat.getOutputCommitter(TeraOutputFormat.java:152)
> at org.apache.hadoop.mapred.Task.initialize(Task.java:606)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request:
> error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify
> failed
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1116)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1066)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4368)
> at
> com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5129)
> at
> com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:5103)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4352)
> at
>
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification
hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#issuecomment-519353802 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 14 | https://github.com/apache/hadoop/pull/575 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/575 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/7/console | | versions | git=2.7.4 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification
hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#issuecomment-515493257 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 12 | https://github.com/apache/hadoop/pull/575 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/575 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/5/console | | versions | git=2.7.4 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification
hadoop-yetus removed a comment on issue #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#issuecomment-475830875 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 25 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 6 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 21 | Maven dependency ordering for branch | | +1 | mvninstall | 966 | trunk passed | | +1 | compile | 991 | trunk passed | | +1 | checkstyle | 199 | trunk passed | | +1 | mvnsite | 285 | trunk passed | | +1 | shadedclient | 1164 | branch has no errors when building and testing our client artifacts. | | +1 | findbugs | 295 | trunk passed | | +1 | javadoc | 172 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 22 | Maven dependency ordering for patch | | +1 | mvninstall | 159 | the patch passed | | +1 | compile | 880 | the patch passed | | -1 | javac | 880 | root generated 2 new + 1483 unchanged - 0 fixed = 1485 total (was 1483) | | -0 | checkstyle | 233 | root: The patch generated 4 new + 32 unchanged - 3 fixed = 36 total (was 35) | | +1 | mvnsite | 243 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 3 | The patch has no ill-formed XML file. | | +1 | shadedclient | 674 | patch has no errors when building and testing our client artifacts. | | +1 | findbugs | 354 | the patch passed | | +1 | javadoc | 154 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 509 | hadoop-common in the patch passed. | | -1 | unit | 4737 | hadoop-hdfs in the patch failed. | | +1 | unit | 292 | hadoop-aws in the patch passed. | | +1 | unit | 66 | hadoop-azure-datalake in the patch passed. | | +1 | asflicense | 55 | The patch does not generate ASF License warnings. | | | | 12300 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes | | | hadoop.hdfs.web.TestWebHdfsTimeouts | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-575/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/575 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux faa498435078 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 73f7b04 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/2/artifact/out/diff-compile-javac-root.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/2/artifact/out/diff-checkstyle-root.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/2/testReport/ | | Max. process+thread count | 4987 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure-datalake U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-575/2/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #575: HADOOP-13327 Output Stream Specification
steveloughran closed pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #575: HADOOP-13327 Output Stream Specification
steveloughran commented on issue #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#issuecomment-549930986 I'm closing this and opening up a peer without any of the S3A changes, #1694. It currently addresses most of the comments of people's reviews, excluding Sean's request about restructuring the model > Could this be in a section about visibility and not in the model definition? Maybe later. "here's the model, here's how that model works with creation, here's how it works when reading/writing" flows much better and visibility should be in that third part. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16676) Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to branch-3.2)
[ https://issues.apache.org/jira/browse/HADOOP-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HADOOP-16676: Attachment: HADOOP-16676.branch-3.2.001.patch Status: Patch Available (was: Open) Uploaded branch-3.2 patch. Compilation passed locally after reverting YARN-9949 (irrelevant to this patch, but breaks compilation). > Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to > branch-3.2) > - > > Key: HADOOP-16676 > URL: https://issues.apache.org/jira/browse/HADOOP-16676 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.1 >Reporter: DW >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16676.branch-3.2.001.patch > > > Hello, > > org.apache.hadoop:hadoop-common define the dependency to jetty-webapp and > jetty-xml in version v9.3.24 with known CVE-2017-9735. Please can you upgrade > to version 9.4.7 or higher? > +--- org.apache.hadoop:hadoop-client:3.2.1 > | +--- org.apache.hadoop:hadoop-common:3.2.1 > | +--- org.eclipse.jetty:jetty-webapp:9.3.24.v20180605 > | | | +--- org.eclipse.jetty:jetty-xml:9.3.24.v20180605 > | | | \--- org.eclipse.jetty:jetty-servlet:9.3.24.v20180605 (*) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342696796 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's Output Streams +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported it's methods, but this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS have +A new interface, `StreamCapabilities` has been implemented to allow callers to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +* HDFS's primary stream implementation is +`org.apache.hadoop.hdfs.DFSOutputStream`. +* The subclass `org.apache.hadoop.hdfs.DFSStripedOutputStream` supports erasure +coding: it removes the `Syncable` behaviors from the base class. +* The output streams `org.apache.hadoop.fs.FSOutputSummer` and +`org.apache.hadoop.fs.ChecksumFileSystem.ChecksumFSOutputSummer` +contain the underlying checksummed output stream used by +both HDFS and the "file" filesystems. + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte]` +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path` can be tracked to form a triple +`Path, open, buffer` + + +```python +Stream = (path, open, buffer) +``` + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path must match that of +`buffer`. That is, the following condition holds: + +```python +FS'.Files(path) == buffer Review comment: yes. Actually pulled over the definition fom the later close() call ``` FS' = FS where data(FS', path) == buffer ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1611: HADOOP-16612 Track Azure Blob File System client-perceived latency
goiri commented on a change in pull request #1611: HADOOP-16612 Track Azure
Blob File System client-perceived latency
URL: https://github.com/apache/hadoop/pull/1611#discussion_r342693643
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -686,210 +784,292 @@ public void setOwner(final Path path, final String
owner, final String group) th
AzureBlobFileSystemException {
if (!getIsNamespaceEnabled()) {
throw new UnsupportedOperationException(
- "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
+ "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
}
-LOG.debug(
-"setOwner filesystem: {} path: {} owner: {} group: {}",
-client.getFileSystem(),
-path.toString(),
-owner,
-group);
+try (AbfsPerfInfo perfInfo = startTracking("setOwner", "setOwner")) {
+
+ LOG.debug(
+ "setOwner filesystem: {} path: {} owner: {} group: {}",
+ client.getFileSystem(),
+ path.toString(),
+ owner,
+ group);
+
+ final String transformedOwner =
identityTransformer.transformUserOrGroupForSetRequest(owner);
+ final String transformedGroup =
identityTransformer.transformUserOrGroupForSetRequest(group);
-final String transformedOwner =
identityTransformer.transformUserOrGroupForSetRequest(owner);
-final String transformedGroup =
identityTransformer.transformUserOrGroupForSetRequest(group);
+ final AbfsRestOperation op = client.setOwner(
+ AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path, true),
+ transformedOwner,
+ transformedGroup);
-client.setOwner(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path,
true), transformedOwner, transformedGroup);
+ perfInfo.registerResult(op.getResult()).registerSuccess(true);
+}
}
public void setPermission(final Path path, final FsPermission permission)
throws
AzureBlobFileSystemException {
if (!getIsNamespaceEnabled()) {
throw new UnsupportedOperationException(
- "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
+ "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
Review comment:
This was correct before. Avoid.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1611: HADOOP-16612 Track Azure Blob File System client-perceived latency
goiri commented on a change in pull request #1611: HADOOP-16612 Track Azure
Blob File System client-perceived latency
URL: https://github.com/apache/hadoop/pull/1611#discussion_r342694985
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##
@@ -471,6 +475,10 @@ public boolean isUpnUsed() {
return this.useUpn;
}
+ public boolean shouldTrackLatency() {
Review comment:
Javadoc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1611: HADOOP-16612 Track Azure Blob File System client-perceived latency
goiri commented on a change in pull request #1611: HADOOP-16612 Track Azure
Blob File System client-perceived latency
URL: https://github.com/apache/hadoop/pull/1611#discussion_r342694260
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -686,210 +784,292 @@ public void setOwner(final Path path, final String
owner, final String group) th
AzureBlobFileSystemException {
if (!getIsNamespaceEnabled()) {
throw new UnsupportedOperationException(
- "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
+ "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
}
-LOG.debug(
-"setOwner filesystem: {} path: {} owner: {} group: {}",
-client.getFileSystem(),
-path.toString(),
-owner,
-group);
+try (AbfsPerfInfo perfInfo = startTracking("setOwner", "setOwner")) {
+
+ LOG.debug(
+ "setOwner filesystem: {} path: {} owner: {} group: {}",
+ client.getFileSystem(),
+ path.toString(),
+ owner,
+ group);
+
+ final String transformedOwner =
identityTransformer.transformUserOrGroupForSetRequest(owner);
+ final String transformedGroup =
identityTransformer.transformUserOrGroupForSetRequest(group);
-final String transformedOwner =
identityTransformer.transformUserOrGroupForSetRequest(owner);
-final String transformedGroup =
identityTransformer.transformUserOrGroupForSetRequest(group);
+ final AbfsRestOperation op = client.setOwner(
+ AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path, true),
+ transformedOwner,
+ transformedGroup);
-client.setOwner(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path,
true), transformedOwner, transformedGroup);
+ perfInfo.registerResult(op.getResult()).registerSuccess(true);
+}
}
public void setPermission(final Path path, final FsPermission permission)
throws
AzureBlobFileSystemException {
if (!getIsNamespaceEnabled()) {
throw new UnsupportedOperationException(
- "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
+ "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
}
-LOG.debug(
-"setPermission filesystem: {} path: {} permission: {}",
-client.getFileSystem(),
-path.toString(),
-permission.toString());
-client.setPermission(AbfsHttpConstants.FORWARD_SLASH +
getRelativePath(path, true),
-String.format(AbfsHttpConstants.PERMISSION_FORMAT,
permission.toOctal()));
+try (AbfsPerfInfo perfInfo = startTracking("setPermission",
"setPermission")) {
+
+ LOG.debug(
+ "setPermission filesystem: {} path: {} permission: {}",
+ client.getFileSystem(),
+ path.toString(),
+ permission.toString());
+
+ final AbfsRestOperation op = client.setPermission(
+ AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path, true),
+ String.format(AbfsHttpConstants.PERMISSION_FORMAT,
permission.toOctal()));
+
+ perfInfo.registerResult(op.getResult()).registerSuccess(true);
+}
}
public void modifyAclEntries(final Path path, final List aclSpec)
throws
AzureBlobFileSystemException {
if (!getIsNamespaceEnabled()) {
throw new UnsupportedOperationException(
- "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
+ "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
}
-LOG.debug(
-"modifyAclEntries filesystem: {} path: {} aclSpec: {}",
-client.getFileSystem(),
-path.toString(),
-AclEntry.aclSpecToString(aclSpec));
+try (AbfsPerfInfo perfInfoGet = startTracking("modifyAclEntries",
"getAclStatus")) {
+
+ LOG.debug(
+ "modifyAclEntries filesystem: {} path: {} aclSpec: {}",
+ client.getFileSystem(),
+ path.toString(),
+ AclEntry.aclSpecToString(aclSpec));
+
+ identityTransformer.transformAclEntriesForSetRequest(aclSpec);
+ final Map modifyAclEntries =
AbfsAclHelper.deserializeAclSpec(AclEntry.aclSpecToString(aclSpec));
+ boolean useUpn = AbfsAclHelper.isUpnFormatAclEntries(modifyAclEntries);
-identityTransformer.transformAclEntriesForSetRequest(aclSpec);
-final Map modifyAclEntries =
AbfsAclHelper.deserializeAclSpec(AclEntry.aclSpecToString(aclSpec));
-boolean useUpn = AbfsAclHelper.isUpnFormatAclEntries(modifyAclEntries);
+ final AbfsRestOperation op =
[GitHub] [hadoop] goiri commented on a change in pull request #1611: HADOOP-16612 Track Azure Blob File System client-perceived latency
goiri commented on a change in pull request #1611: HADOOP-16612 Track Azure
Blob File System client-perceived latency
URL: https://github.com/apache/hadoop/pull/1611#discussion_r342694706
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##
@@ -121,7 +121,17 @@ public AbfsHttpOperation getResult() {
* HTTP operations.
*/
void execute() throws AzureBlobFileSystemException {
+// see if we have latency reports from the previous requests
+String latencyHeader = this.client.getAbfsPerfTracker().getClientLatency();
+
+if (latencyHeader != null && !latencyHeader.isEmpty()) {
+ AbfsHttpHeader httpHeader =
+ new
AbfsHttpHeader(HttpHeaderConfigurations.X_MS_ABFS_CLIENT_LATENCY,
latencyHeader);
+ requestHeaders.add(httpHeader);
+}
+
int retryCount = 0;
+
Review comment:
Avoid.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1611: HADOOP-16612 Track Azure Blob File System client-perceived latency
goiri commented on a change in pull request #1611: HADOOP-16612 Track Azure
Blob File System client-perceived latency
URL: https://github.com/apache/hadoop/pull/1611#discussion_r342693741
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -686,210 +784,292 @@ public void setOwner(final Path path, final String
owner, final String group) th
AzureBlobFileSystemException {
if (!getIsNamespaceEnabled()) {
throw new UnsupportedOperationException(
- "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
+ "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
}
-LOG.debug(
-"setOwner filesystem: {} path: {} owner: {} group: {}",
-client.getFileSystem(),
-path.toString(),
-owner,
-group);
+try (AbfsPerfInfo perfInfo = startTracking("setOwner", "setOwner")) {
+
+ LOG.debug(
+ "setOwner filesystem: {} path: {} owner: {} group: {}",
+ client.getFileSystem(),
+ path.toString(),
+ owner,
+ group);
+
+ final String transformedOwner =
identityTransformer.transformUserOrGroupForSetRequest(owner);
+ final String transformedGroup =
identityTransformer.transformUserOrGroupForSetRequest(group);
-final String transformedOwner =
identityTransformer.transformUserOrGroupForSetRequest(owner);
-final String transformedGroup =
identityTransformer.transformUserOrGroupForSetRequest(group);
+ final AbfsRestOperation op = client.setOwner(
+ AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path, true),
+ transformedOwner,
+ transformedGroup);
-client.setOwner(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path,
true), transformedOwner, transformedGroup);
+ perfInfo.registerResult(op.getResult()).registerSuccess(true);
+}
}
public void setPermission(final Path path, final FsPermission permission)
throws
AzureBlobFileSystemException {
if (!getIsNamespaceEnabled()) {
throw new UnsupportedOperationException(
- "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
+ "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
}
-LOG.debug(
-"setPermission filesystem: {} path: {} permission: {}",
-client.getFileSystem(),
-path.toString(),
-permission.toString());
-client.setPermission(AbfsHttpConstants.FORWARD_SLASH +
getRelativePath(path, true),
-String.format(AbfsHttpConstants.PERMISSION_FORMAT,
permission.toOctal()));
+try (AbfsPerfInfo perfInfo = startTracking("setPermission",
"setPermission")) {
+
+ LOG.debug(
+ "setPermission filesystem: {} path: {} permission: {}",
+ client.getFileSystem(),
+ path.toString(),
+ permission.toString());
+
+ final AbfsRestOperation op = client.setPermission(
+ AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path, true),
+ String.format(AbfsHttpConstants.PERMISSION_FORMAT,
permission.toOctal()));
+
+ perfInfo.registerResult(op.getResult()).registerSuccess(true);
+}
}
public void modifyAclEntries(final Path path, final List aclSpec)
throws
AzureBlobFileSystemException {
if (!getIsNamespaceEnabled()) {
throw new UnsupportedOperationException(
- "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
+ "This operation is only valid for storage accounts with the
hierarchical namespace enabled.");
Review comment:
Revert.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1611: HADOOP-16612 Track Azure Blob File System client-perceived latency
goiri commented on a change in pull request #1611: HADOOP-16612 Track Azure
Blob File System client-perceived latency
URL: https://github.com/apache/hadoop/pull/1611#discussion_r342694828
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##
@@ -121,7 +121,17 @@ public AbfsHttpOperation getResult() {
* HTTP operations.
*/
void execute() throws AzureBlobFileSystemException {
+// see if we have latency reports from the previous requests
+String latencyHeader = this.client.getAbfsPerfTracker().getClientLatency();
+
Review comment:
To split in two blocks, I would not have this break line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342694675 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's Output Streams +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported it's methods, but this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS have +A new interface, `StreamCapabilities` has been implemented to allow callers to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +* HDFS's primary stream implementation is +`org.apache.hadoop.hdfs.DFSOutputStream`. +* The subclass `org.apache.hadoop.hdfs.DFSStripedOutputStream` supports erasure +coding: it removes the `Syncable` behaviors from the base class. +* The output streams `org.apache.hadoop.fs.FSOutputSummer` and +`org.apache.hadoop.fs.ChecksumFileSystem.ChecksumFSOutputSummer` +contain the underlying checksummed output stream used by +both HDFS and the "file" filesystems. + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte]` +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path` can be tracked to form a triple +`Path, open, buffer` + + +```python +Stream = (path, open, buffer) +``` + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path must match that of +`buffer`. That is, the following condition holds: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path will see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability +guarantees, not visibility of data. + + +### State of Stream and filesystem after `Filesystem.create()` + + +The output stream returned by a `FileSystem.create(path)` call contains no +data: + +```python +Stream' = (path, true, []) Review comment: its how it can be modelled after the call. I'll add a ref in filesystem.md This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342694209 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's Output Streams +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported it's methods, but this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS have +A new interface, `StreamCapabilities` has been implemented to allow callers to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +* HDFS's primary stream implementation is +`org.apache.hadoop.hdfs.DFSOutputStream`. +* The subclass `org.apache.hadoop.hdfs.DFSStripedOutputStream` supports erasure +coding: it removes the `Syncable` behaviors from the base class. +* The output streams `org.apache.hadoop.fs.FSOutputSummer` and +`org.apache.hadoop.fs.ChecksumFileSystem.ChecksumFSOutputSummer` +contain the underlying checksummed output stream used by +both HDFS and the "file" filesystems. + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte]` +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path` can be tracked to form a triple +`Path, open, buffer` + + +```python +Stream = (path, open, buffer) +``` + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path must match that of +`buffer`. That is, the following condition holds: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path will see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability +guarantees, not visibility of data. + + +### State of Stream and filesystem after `Filesystem.create()` + + +The output stream returned by a `FileSystem.create(path)` call contains no +data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` must contain a 0-byte file at the path: Review comment: +added the " within a filesystem `FS`" to the create and append references This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16676) Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to branch-3.2)
[ https://issues.apache.org/jira/browse/HADOOP-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HADOOP-16676: Target Version/s: 3.2.2 > Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to > branch-3.2) > - > > Key: HADOOP-16676 > URL: https://issues.apache.org/jira/browse/HADOOP-16676 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.1 >Reporter: DW >Assignee: Siyao Meng >Priority: Major > > Hello, > > org.apache.hadoop:hadoop-common define the dependency to jetty-webapp and > jetty-xml in version v9.3.24 with known CVE-2017-9735. Please can you upgrade > to version 9.4.7 or higher? > +--- org.apache.hadoop:hadoop-client:3.2.1 > | +--- org.apache.hadoop:hadoop-common:3.2.1 > | +--- org.eclipse.jetty:jetty-webapp:9.3.24.v20180605 > | | | +--- org.eclipse.jetty:jetty-xml:9.3.24.v20180605 > | | | \--- org.eclipse.jetty:jetty-servlet:9.3.24.v20180605 (*) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16676) Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to branch-3.2)
[ https://issues.apache.org/jira/browse/HADOOP-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HADOOP-16676: Summary: Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to branch-3.2) (was: Security Vulnerability for dependency jetty-xml -please upgrade) > Security Vulnerability for dependency jetty-xml (Backport HADOOP-16152 to > branch-3.2) > - > > Key: HADOOP-16676 > URL: https://issues.apache.org/jira/browse/HADOOP-16676 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.1 >Reporter: DW >Assignee: Siyao Meng >Priority: Major > > Hello, > > org.apache.hadoop:hadoop-common define the dependency to jetty-webapp and > jetty-xml in version v9.3.24 with known CVE-2017-9735. Please can you upgrade > to version 9.4.7 or higher? > +--- org.apache.hadoop:hadoop-client:3.2.1 > | +--- org.apache.hadoop:hadoop-common:3.2.1 > | +--- org.eclipse.jetty:jetty-webapp:9.3.24.v20180605 > | | | +--- org.eclipse.jetty:jetty-xml:9.3.24.v20180605 > | | | \--- org.eclipse.jetty:jetty-servlet:9.3.24.v20180605 (*) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342691978 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's Output Streams +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported it's methods, but this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS have +A new interface, `StreamCapabilities` has been implemented to allow callers to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +* HDFS's primary stream implementation is +`org.apache.hadoop.hdfs.DFSOutputStream`. +* The subclass `org.apache.hadoop.hdfs.DFSStripedOutputStream` supports erasure +coding: it removes the `Syncable` behaviors from the base class. +* The output streams `org.apache.hadoop.fs.FSOutputSummer` and +`org.apache.hadoop.fs.ChecksumFileSystem.ChecksumFSOutputSummer` +contain the underlying checksummed output stream used by +both HDFS and the "file" filesystems. + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte]` +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path` can be tracked to form a triple +`Path, open, buffer` + + +```python +Stream = (path, open, buffer) +``` + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path must match that of +`buffer`. That is, the following condition holds: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path will see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability +guarantees, not visibility of data. + + +### State of Stream and filesystem after `Filesystem.create()` + + +The output stream returned by a `FileSystem.create(path)` call contains no +data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` must contain a 0-byte file at the path: Review comment: well, in `FS` there was no file This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342691729 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's Output Streams +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported it's methods, but this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS have +A new interface, `StreamCapabilities` has been implemented to allow callers to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +* HDFS's primary stream implementation is +`org.apache.hadoop.hdfs.DFSOutputStream`. +* The subclass `org.apache.hadoop.hdfs.DFSStripedOutputStream` supports erasure +coding: it removes the `Syncable` behaviors from the base class. +* The output streams `org.apache.hadoop.fs.FSOutputSummer` and +`org.apache.hadoop.fs.ChecksumFileSystem.ChecksumFSOutputSummer` +contain the underlying checksummed output stream used by +both HDFS and the "file" filesystems. + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte]` +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path` can be tracked to form a triple +`Path, open, buffer` + + +```python +Stream = (path, open, buffer) +``` + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path must match that of +`buffer`. That is, the following condition holds: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path will see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16676) Security Vulnerability for dependency jetty-xml -please upgrade
[ https://issues.apache.org/jira/browse/HADOOP-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng reassigned HADOOP-16676: --- Assignee: Siyao Meng > Security Vulnerability for dependency jetty-xml -please upgrade > --- > > Key: HADOOP-16676 > URL: https://issues.apache.org/jira/browse/HADOOP-16676 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.1 >Reporter: DW >Assignee: Siyao Meng >Priority: Major > > Hello, > > org.apache.hadoop:hadoop-common define the dependency to jetty-webapp and > jetty-xml in version v9.3.24 with known CVE-2017-9735. Please can you upgrade > to version 9.4.7 or higher? > +--- org.apache.hadoop:hadoop-client:3.2.1 > | +--- org.apache.hadoop:hadoop-common:3.2.1 > | +--- org.eclipse.jetty:jetty-webapp:9.3.24.v20180605 > | | | +--- org.eclipse.jetty:jetty-xml:9.3.24.v20180605 > | | | \--- org.eclipse.jetty:jetty-servlet:9.3.24.v20180605 (*) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342689221 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other Review comment: added details on close() in both that method and on an implementor's notes section. Found a JIRA where the Accumulo team "expressed concern": This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342676604 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's Output Streams +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported it's methods, but this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS have Review comment: fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output
Stream Specification
URL: https://github.com/apache/hadoop/pull/575#discussion_r342676272
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/StreamStateModel.java
##
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.io.IOException;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+
+import static org.apache.hadoop.fs.FSExceptionMessages.STREAM_IS_CLOSED;
+
+/**
+ * Models a stream's state and can be used for checking this state before
+ * any operation.
+ *
+ * The model has three states: Open, Error, and Closed,
+ *
+ *
+ * Open: caller can interact with the stream.
+ * Error: all operations will raise the previously recorded exception.
+ * Closed: operations will be rejected.
+ *
+ */
+public class StreamStateModel {
+
+ /**
+ * States of the stream.
+ */
+ public enum State {
+
+/**
+ * Stream is open.
+ */
+Open,
+
+/**
+ * Stream is in an error state.
+ * It is not expected to recover from this.
+ */
+Error,
+
+/**
+ * Stream is now closed. Operations will fail.
+ */
+Closed
+ }
+
+ /**
+ * Path; if not empty then a {@link PathIOException} will be raised
+ * containing this path.
+ */
+ private final String path;
+
+ /** Lock. Not considering an InstrumentedWriteLock, but it is an option. */
+ private final Lock lock = new ReentrantLock();
+
+ /**
+ * Initial state: open.
+ * This is volatile: it can be queried without encountering any locks.
+ * However, to guarantee the state is constant through the life of an
+ * operation, updates must be through the synchronized methods.
+ */
+ private volatile State state = State.Open;
+
+ /** Any exception to raise on the next checkOpen call. */
Review comment:
cut class
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342674765 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's Output Streams +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported it's methods, but this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS have +A new interface, `StreamCapabilities` has been implemented to allow callers to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +* HDFS's primary stream implementation is +`org.apache.hadoop.hdfs.DFSOutputStream`. +* The subclass `org.apache.hadoop.hdfs.DFSStripedOutputStream` supports erasure +coding: it removes the `Syncable` behaviors from the base class. +* The output streams `org.apache.hadoop.fs.FSOutputSummer` and +`org.apache.hadoop.fs.ChecksumFileSystem.ChecksumFSOutputSummer` +contain the underlying checksummed output stream used by +both HDFS and the "file" filesystems. + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte]` +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path` can be tracked to form a triple +`Path, open, buffer` + + +```python +Stream = (path, open, buffer) +``` + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path must match that of +`buffer`. That is, the following condition holds: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path will see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability +guarantees, not visibility of data. + + +### State of Stream and filesystem after `Filesystem.create()` + + +The output stream returned by a `FileSystem.create(path)` call contains no +data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` must contain a 0-byte file at the path: + +```python +data(FS', path) == [] +``` + +Accordingly, the the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: the 0-byte empty file may not exist in the filesystem. + +### State of Stream and filesystem after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in the `close()` operation, or at any other +time the stream chooses to do so, the contents of the file +are replaced with the current buffer + +```python +Stream' = (path, true, buffer) +FS' = FS where data(FS', path) == buffer +``` + +After a call to `close()`, the stream is closed for all operations other +than `close()`; they MAY fail with `IOException` or `RuntimeException`. + +```python +Stream' = (path, false, []) +``` + +The `close()` operation must be idempotent with the sole attempt to write the +data made in the first invocation. + +1. If `close()` succeeds, subsequent calls are no-ops. +1. If `close()` fails, again, subsequent calls are no-ops. They MAY rethrow +the previous exception, but they MUST NOT retry the write. + + + + + +# Class `FSDataOutputStream` + + +```java +public class FSDataOutputStream +
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342674927 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's Output Streams +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported it's methods, but this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS have +A new interface, `StreamCapabilities` has been implemented to allow callers to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +* HDFS's primary stream implementation is +`org.apache.hadoop.hdfs.DFSOutputStream`. +* The subclass `org.apache.hadoop.hdfs.DFSStripedOutputStream` supports erasure +coding: it removes the `Syncable` behaviors from the base class. +* The output streams `org.apache.hadoop.fs.FSOutputSummer` and +`org.apache.hadoop.fs.ChecksumFileSystem.ChecksumFSOutputSummer` +contain the underlying checksummed output stream used by +both HDFS and the "file" filesystems. + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte]` +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path` can be tracked to form a triple +`Path, open, buffer` + + +```python +Stream = (path, open, buffer) +``` + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path must match that of +`buffer`. That is, the following condition holds: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path will see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability +guarantees, not visibility of data. + + +### State of Stream and filesystem after `Filesystem.create()` + + +The output stream returned by a `FileSystem.create(path)` call contains no +data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` must contain a 0-byte file at the path: + +```python +data(FS', path) == [] +``` + +Accordingly, the the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: the 0-byte empty file may not exist in the filesystem. + +### State of Stream and filesystem after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in the `close()` operation, or at any other +time the stream chooses to do so, the contents of the file +are replaced with the current buffer + +```python +Stream' = (path, true, buffer) +FS' = FS where data(FS', path) == buffer +``` + +After a call to `close()`, the stream is closed for all operations other +than `close()`; they MAY fail with `IOException` or `RuntimeException`. + +```python +Stream' = (path, false, []) +``` + +The `close()` operation must be idempotent with the sole attempt to write the +data made in the first invocation. + +1. If `close()` succeeds, subsequent calls are no-ops. +1. If `close()` fails, again, subsequent calls are no-ops. They MAY rethrow +the previous exception, but they MUST NOT retry the write. + + + + + +# Class `FSDataOutputStream` + + +```java +public class FSDataOutputStream +
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342674113 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's Output Streams +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported it's methods, but this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS have +A new interface, `StreamCapabilities` has been implemented to allow callers to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +* HDFS's primary stream implementation is +`org.apache.hadoop.hdfs.DFSOutputStream`. +* The subclass `org.apache.hadoop.hdfs.DFSStripedOutputStream` supports erasure +coding: it removes the `Syncable` behaviors from the base class. +* The output streams `org.apache.hadoop.fs.FSOutputSummer` and +`org.apache.hadoop.fs.ChecksumFileSystem.ChecksumFSOutputSummer` +contain the underlying checksummed output stream used by +both HDFS and the "file" filesystems. + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte]` +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path` can be tracked to form a triple +`Path, open, buffer` + + +```python +Stream = (path, open, buffer) +``` + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path must match that of +`buffer`. That is, the following condition holds: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path will see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability +guarantees, not visibility of data. + + +### State of Stream and filesystem after `Filesystem.create()` + + +The output stream returned by a `FileSystem.create(path)` call contains no +data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` must contain a 0-byte file at the path: + +```python +data(FS', path) == [] +``` + +Accordingly, the the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: the 0-byte empty file may not exist in the filesystem. + +### State of Stream and filesystem after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in the `close()` operation, or at any other +time the stream chooses to do so, the contents of the file +are replaced with the current buffer + +```python +Stream' = (path, true, buffer) +FS' = FS where data(FS', path) == buffer +``` + +After a call to `close()`, the stream is closed for all operations other +than `close()`; they MAY fail with `IOException` or `RuntimeException`. + +```python +Stream' = (path, false, []) +``` + +The `close()` operation must be idempotent with the sole attempt to write the +data made in the first invocation. + +1. If `close()` succeeds, subsequent calls are no-ops. +1. If `close()` fails, again, subsequent calls are no-ops. They MAY rethrow +the previous exception, but they MUST NOT retry the write. + + + + + +# Class `FSDataOutputStream` + + +```java +public class FSDataOutputStream +
[jira] [Commented] (HADOOP-16683) Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException
[
https://issues.apache.org/jira/browse/HADOOP-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967666#comment-16967666
]
Peter Bacsko commented on HADOOP-16683:
---
[~adam.antal] I think the return value of this function should be {{boolean}}
instead of {{Throwable}}, because you don't really do much with the returned
object:
{{private static Throwable getWrappedAccessControlException(Exception e)}}
> Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped
> AccessControlException
> --
>
> Key: HADOOP-16683
> URL: https://issues.apache.org/jira/browse/HADOOP-16683
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: HADOOP-16683.001.patch
>
>
> Follow up patch on HADOOP-16580.
> We successfully disabled the retry in case of an AccessControlException which
> has resolved some of the cases, but in other cases AccessControlException is
> wrapped inside another IOException and you can only get the original
> exception by calling getCause().
> Let's add this extra case as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342665757 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's Output Streams +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported it's methods, but this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS have +A new interface, `StreamCapabilities` has been implemented to allow callers to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +* HDFS's primary stream implementation is +`org.apache.hadoop.hdfs.DFSOutputStream`. +* The subclass `org.apache.hadoop.hdfs.DFSStripedOutputStream` supports erasure +coding: it removes the `Syncable` behaviors from the base class. +* The output streams `org.apache.hadoop.fs.FSOutputSummer` and +`org.apache.hadoop.fs.ChecksumFileSystem.ChecksumFSOutputSummer` +contain the underlying checksummed output stream used by +both HDFS and the "file" filesystems. + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte]` +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path` can be tracked to form a triple +`Path, open, buffer` + + +```python +Stream = (path, open, buffer) +``` + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path must match that of +`buffer`. That is, the following condition holds: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path will see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability +guarantees, not visibility of data. + + +### State of Stream and filesystem after `Filesystem.create()` + + +The output stream returned by a `FileSystem.create(path)` call contains no +data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` must contain a 0-byte file at the path: + +```python +data(FS', path) == [] +``` + +Accordingly, the the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: the 0-byte empty file may not exist in the filesystem. + +### State of Stream and filesystem after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in the `close()` operation, or at any other +time the stream chooses to do so, the contents of the file +are replaced with the current buffer + +```python +Stream' = (path, true, buffer) +FS' = FS where data(FS', path) == buffer +``` + +After a call to `close()`, the stream is closed for all operations other +than `close()`; they MAY fail with `IOException` or `RuntimeException`. + +```python +Stream' = (path, false, []) +``` + +The `close()` operation must be idempotent with the sole attempt to write the +data made in the first invocation. + +1. If `close()` succeeds, subsequent calls are no-ops. +1. If `close()` fails, again, subsequent calls are no-ops. They MAY rethrow +the previous exception, but they MUST NOT retry the write. + + + + + +# Class `FSDataOutputStream` + + +```java +public class FSDataOutputStream +
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342662761 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's Output Streams +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported it's methods, but this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS have +A new interface, `StreamCapabilities` has been implemented to allow callers to probe the exact capabilities of a stream, even transitively Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output
Stream Specification
URL: https://github.com/apache/hadoop/pull/575#discussion_r342661113
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/StreamStateModel.java
##
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.impl;
+
+import java.io.IOException;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+import com.google.common.base.Preconditions;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+
+import static org.apache.hadoop.fs.FSExceptionMessages.STREAM_IS_CLOSED;
+
+/**
+ * Models a stream's state and can be used for checking this state before
+ * any operation.
+ *
+ * The model has three states: Open, Error, and Closed,
+ *
+ *
+ * Open: caller can interact with the stream.
+ * Error: all operations will raise the previously recorded exception.
+ * Closed: operations will be rejected.
+ *
+ */
+public class StreamStateModel {
+
+ /**
+ * States of the stream.
+ */
+ public enum State {
+
+/**
+ * Stream is open.
+ */
+Open,
+
+/**
+ * Stream is in an error state.
+ * It is not expected to recover from this.
+ */
+Error,
+
+/**
+ * Stream is now closed. Operations will fail.
+ */
+Closed
+ }
+
+ /**
+ * Path; if not empty then a {@link PathIOException} will be raised
+ * containing this path.
+ */
+ private final String path;
+
+ /** Lock. Not considering an InstrumentedWriteLock, but it is an option. */
+ private final Lock lock = new ReentrantLock();
+
+ /**
+ * Initial state: open.
+ * This is volatile: it can be queried without encountering any locks.
+ * However, to guarantee the state is constant through the life of an
+ * operation, updates must be through the synchronized methods.
+ */
+ private volatile State state = State.Open;
+
+ /** Any exception to raise on the next checkOpen call. */
+ private IOException exception;
+
+ public StreamStateModel(final Path path) {
+this.path = path.toString();
+ }
+
+ public StreamStateModel(final String path) {
+this.path = path;
+ }
+
+ /**
+ * Get the current state.
+ * Not synchronized; lock if you want consistency across calls.
+ * @return the current state.
+ */
+ public State getState() {
+return state;
+ }
+
+ /**
+ * Change state to closed. No-op if the state was in closed or error
+ * @return true if the state changed.
+ */
+ public synchronized boolean enterClosedState() {
+if (state == State.Open) {
+ state = State.Closed;
+ return true;
+} else {
+ return false;
+}
+ }
+
+ /**
+ * Change state to error and stores first error so it can be re-thrown.
+ * If already in error: return previous exception.
+ * @param ex the exception to record
+ * @return the exception set when the error state was entered.
+ */
+ public synchronized IOException enterErrorState(final IOException ex) {
+Preconditions.checkArgument(ex != null, "Null exception");
+switch (state) {
+ // a stream can go into the error state when open or closed
+case Open:
+case Closed:
+ exception = ex;
+ state = State.Error;
+ break;
+case Error:
+ // already in this state; retain the previous exception.
+ break;
+}
+return exception;
+ }
+
+ /**
+ * Check a stream is open.
+ * If in an error state: rethrow that exception. If closed,
+ * throw an exception about that.
+ * @throws IOException if the stream is not open.
+ */
+ public synchronized void checkOpen() throws IOException {
+switch (state) {
+case Open:
+ return;
+
+case Error:
+ throw exception;
+
+case Closed:
+ if (StringUtils.isNotEmpty(path)) {
+throw new PathIOException(path, STREAM_IS_CLOSED);
+ } else {
+throw new IOException(STREAM_IS_CLOSED);
+ }
+}
+ }
+
+ /**
+ * Acquire an exclusive lock.
+ * @param checkOpen must the stream be open?
+ * @throws
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342665269 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's Output Streams +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported it's methods, but this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS have +A new interface, `StreamCapabilities` has been implemented to allow callers to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +* HDFS's primary stream implementation is +`org.apache.hadoop.hdfs.DFSOutputStream`. +* The subclass `org.apache.hadoop.hdfs.DFSStripedOutputStream` supports erasure +coding: it removes the `Syncable` behaviors from the base class. +* The output streams `org.apache.hadoop.fs.FSOutputSummer` and +`org.apache.hadoop.fs.ChecksumFileSystem.ChecksumFSOutputSummer` +contain the underlying checksummed output stream used by +both HDFS and the "file" filesystems. + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte]` +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path` can be tracked to form a triple +`Path, open, buffer` + + +```python +Stream = (path, open, buffer) +``` + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path must match that of +`buffer`. That is, the following condition holds: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path will see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability +guarantees, not visibility of data. + + +### State of Stream and filesystem after `Filesystem.create()` + + +The output stream returned by a `FileSystem.create(path)` call contains no +data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` must contain a 0-byte file at the path: + +```python +data(FS', path) == [] +``` + +Accordingly, the the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: the 0-byte empty file may not exist in the filesystem. + +### State of Stream and filesystem after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in the `close()` operation, or at any other +time the stream chooses to do so, the contents of the file +are replaced with the current buffer + +```python +Stream' = (path, true, buffer) +FS' = FS where data(FS', path) == buffer +``` + +After a call to `close()`, the stream is closed for all operations other +than `close()`; they MAY fail with `IOException` or `RuntimeException`. + +```python +Stream' = (path, false, []) +``` + +The `close()` operation must be idempotent with the sole attempt to write the +data made in the first invocation. + +1. If `close()` succeeds, subsequent calls are no-ops. +1. If `close()` fails, again, subsequent calls are no-ops. They MAY rethrow +the previous exception, but they MUST NOT retry the write. + + + + + +# Class `FSDataOutputStream` + + +```java +public class FSDataOutputStream +
[GitHub] [hadoop] steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification
steveloughran commented on a change in pull request #575: HADOOP-13327 Output Stream Specification URL: https://github.com/apache/hadoop/pull/575#discussion_r342659129 ## File path: hadoop-common-project/hadoop-common/src/site/markdown/filesystem/outputstream.md ## @@ -0,0 +1,857 @@ + + + + +# Output: `OutputStream`, `Syncable` and `StreamCapabilities` + + +With the exception of `FileSystem.copyFromLocalFile()`, +all API operations which write data to a filesystem in Hadoop do so +through the Java "OutputStreams" API. More specifically, they do +so through `OutputStream` subclasses obtained through calls to +`FileSystem.create()`, `FileSystem.append()`, +or `FSDataOutputStreamBuilder.build()`. + +These all return instances of `FSDataOutputStream`, through which data +can be written through various `write()` methods. +After a stream's `close()` method is called, all data written to the +stream MUST BE persisted to the fileysystem and visible to oll other +clients attempting to read data from that path via `FileSystem.open()`. + +As well as operations to write the data, Hadoop's Output Streams +provide methods to flush buffered data back to the filesystem, +so as to ensure that the data is reliably persisted and/or visible +to other callers. This is done via the `Syncable` interface. It was +originally intended that the presence of this interface could be interpreted +as a guarantee that the stream supported it's methods, but this has proven +impossible to guarantee as the static nature of the interface is incompatible +with filesystems whose syncability semantics may vary on a store/path basis. +As an example, erasure coded files in HDFS have +A new interface, `StreamCapabilities` has been implemented to allow callers to probe the exact capabilities of a stream, even transitively +through a chain of streams. + +* HDFS's primary stream implementation is +`org.apache.hadoop.hdfs.DFSOutputStream`. +* The subclass `org.apache.hadoop.hdfs.DFSStripedOutputStream` supports erasure +coding: it removes the `Syncable` behaviors from the base class. +* The output streams `org.apache.hadoop.fs.FSOutputSummer` and +`org.apache.hadoop.fs.ChecksumFileSystem.ChecksumFSOutputSummer` +contain the underlying checksummed output stream used by +both HDFS and the "file" filesystems. + + +## Output Stream Model + +For this specification, an output stream can be viewed as a list of bytes +stored in in the client + +```python +buffer: List[byte]` +``` + +A flag, `open` tracks whether the stream is open: after the stream +is closed no more data may be written to it: + +```python +open: bool +buffer: List[byte] +``` + +The destination path of the stream, `path` can be tracked to form a triple +`Path, open, buffer` + + +```python +Stream = (path, open, buffer) +``` + +(Immediately) after `Syncable` operations which flush data to the filesystem, +the data at the stream's destination path must match that of +`buffer`. That is, the following condition holds: + +```python +FS'.Files(path) == buffer +``` + +Any client reading the data at the path will see the new data. +The two sync operations, `hflush()` and `hsync()` differ in their durability +guarantees, not visibility of data. + + +### State of Stream and filesystem after `Filesystem.create()` + + +The output stream returned by a `FileSystem.create(path)` call contains no +data: + +```python +Stream' = (path, true, []) +``` + +The filesystem `FS'` must contain a 0-byte file at the path: + +```python +data(FS', path) == [] +``` + +Accordingly, the the initial state of `Stream'.buffer` is implicitly +consistent with the data at the filesystem. + + +*Object Stores*: the 0-byte empty file may not exist in the filesystem. + +### State of Stream and filesystem after `Filesystem.append()` + +The output stream returned from a call of + `FileSystem.append(path, buffersize, progress)`, +can be modelled as a stream whose `buffer` is intialized to that of +the original file: + +```python +Stream' = (path, true, data(FS, path)) +``` + + Persisting data + +When the stream writes data back to its store, be it in any +supported flush operation, in the `close()` operation, or at any other +time the stream chooses to do so, the contents of the file +are replaced with the current buffer + +```python +Stream' = (path, true, buffer) +FS' = FS where data(FS', path) == buffer +``` + +After a call to `close()`, the stream is closed for all operations other +than `close()`; they MAY fail with `IOException` or `RuntimeException`. + +```python +Stream' = (path, false, []) +``` + +The `close()` operation must be idempotent with the sole attempt to write the +data made in the first invocation. + +1. If `close()` succeeds, subsequent calls are no-ops. +1. If `close()` fails, again, subsequent calls are no-ops. They MAY rethrow +the previous exception, but they MUST NOT retry the write. + + + + + +# Class `FSDataOutputStream` + + +```java +public class FSDataOutputStream +
