[jira] [Commented] (HADOOP-15133) [JDK9] Ignore com.sun.javadoc.* and com.sun.tools.* in animal-sniffer-maven-plugin to compile with Java 9
[
https://issues.apache.org/jira/browse/HADOOP-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297989#comment-16297989
]
genericqa commented on HADOOP-15133:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
7s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
26m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
11s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15133 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12902981/HADOOP-15133.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux f286674d120b 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d62932c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13859/testReport/ |
| Max. process+thread count | 312 (vs. ulimit of 5000) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13859/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [JDK9] Ignore com.sun.javadoc.* and com.sun.tools.* in
> animal-sniffer-maven-plugin to compile with Java 9
> -
>
> Key: HADOOP-15133
> URL: https://issues.apache.org/jira/browse/HADOOP-15133
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>
[jira] [Updated] (HADOOP-15133) [JDK9] Ignore com.sun.javadoc.* and com.sun.tools.* in animal-sniffer-maven-plugin to compile with Java 9
[
https://issues.apache.org/jira/browse/HADOOP-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15133:
---
Attachment: HADOOP-15133.001.patch
> [JDK9] Ignore com.sun.javadoc.* and com.sun.tools.* in
> animal-sniffer-maven-plugin to compile with Java 9
> -
>
> Key: HADOOP-15133
> URL: https://issues.apache.org/jira/browse/HADOOP-15133
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Akira Ajisaka
> Attachments: HADOOP-15133.001.patch
>
>
> com.sun.javadoc and com.sun.tools are internal APIs and are not included in
> java18 profile, so signature check fails with JDK9.
> {noformat}
> $ mvn clean install -DskipTests -DskipShade
> (snip)
> [INFO] --- animal-sniffer-maven-plugin:1.16:check (signature-check) @
> hadoop-annotations ---
> [INFO] Checking unresolved references to
> org.codehaus.mojo.signature:java18:1.0
> [ERROR]
> /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java:56:
> Undefined reference: com.sun.javadoc.RootDoc
> (snip)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15133) [JDK9] Ignore com.sun.javadoc.* and com.sun.tools.* in animal-sniffer-maven-plugin to compile with Java 9
[
https://issues.apache.org/jira/browse/HADOOP-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15133:
---
Assignee: Akira Ajisaka
Status: Patch Available (was: Open)
> [JDK9] Ignore com.sun.javadoc.* and com.sun.tools.* in
> animal-sniffer-maven-plugin to compile with Java 9
> -
>
> Key: HADOOP-15133
> URL: https://issues.apache.org/jira/browse/HADOOP-15133
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
> Attachments: HADOOP-15133.001.patch
>
>
> com.sun.javadoc and com.sun.tools are internal APIs and are not included in
> java18 profile, so signature check fails with JDK9.
> {noformat}
> $ mvn clean install -DskipTests -DskipShade
> (snip)
> [INFO] --- animal-sniffer-maven-plugin:1.16:check (signature-check) @
> hadoop-annotations ---
> [INFO] Checking unresolved references to
> org.codehaus.mojo.signature:java18:1.0
> [ERROR]
> /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java:56:
> Undefined reference: com.sun.javadoc.RootDoc
> (snip)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15133) [JDK9] Ignore com.sun.javadoc.* and com.sun.tools.* in animal-sniffer-maven-plugin to compile with Java 9
[
https://issues.apache.org/jira/browse/HADOOP-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-15133:
---
Description:
com.sun.javadoc and com.sun.tools are internal APIs and are not included in
java18 profile, so signature check fails with JDK9.
{noformat}
$ mvn clean install -DskipTests -DskipShade
(snip)
[INFO] --- animal-sniffer-maven-plugin:1.16:check (signature-check) @
hadoop-annotations ---
[INFO] Checking unresolved references to org.codehaus.mojo.signature:java18:1.0
[ERROR]
/Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java:56:
Undefined reference: com.sun.javadoc.RootDoc
(snip)
{noformat}
> [JDK9] Ignore com.sun.javadoc.* and com.sun.tools.* in
> animal-sniffer-maven-plugin to compile with Java 9
> -
>
> Key: HADOOP-15133
> URL: https://issues.apache.org/jira/browse/HADOOP-15133
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Akira Ajisaka
>
> com.sun.javadoc and com.sun.tools are internal APIs and are not included in
> java18 profile, so signature check fails with JDK9.
> {noformat}
> $ mvn clean install -DskipTests -DskipShade
> (snip)
> [INFO] --- animal-sniffer-maven-plugin:1.16:check (signature-check) @
> hadoop-annotations ---
> [INFO] Checking unresolved references to
> org.codehaus.mojo.signature:java18:1.0
> [ERROR]
> /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java:56:
> Undefined reference: com.sun.javadoc.RootDoc
> (snip)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15133) [JDK9] Ignore com.sun.javadoc.* and com.sun.tools.* in animal-sniffer-maven-plugin to compile with Java 9
[ https://issues.apache.org/jira/browse/HADOOP-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-15133: --- Issue Type: Sub-task (was: Bug) Parent: HADOOP-11123 > [JDK9] Ignore com.sun.javadoc.* and com.sun.tools.* in > animal-sniffer-maven-plugin to compile with Java 9 > - > > Key: HADOOP-15133 > URL: https://issues.apache.org/jira/browse/HADOOP-15133 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15133) [JDK9] Ignore com.sun.javadoc.* and com.sun.tools.* in animal-sniffer-maven-plugin to compile with Java 9
Akira Ajisaka created HADOOP-15133: -- Summary: [JDK9] Ignore com.sun.javadoc.* and com.sun.tools.* in animal-sniffer-maven-plugin to compile with Java 9 Key: HADOOP-15133 URL: https://issues.apache.org/jira/browse/HADOOP-15133 Project: Hadoop Common Issue Type: Bug Reporter: Akira Ajisaka -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14965) s3a input stream "normal" fadvise mode to be adaptive
[ https://issues.apache.org/jira/browse/HADOOP-14965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297868#comment-16297868 ] Aaron Fabbri commented on HADOOP-14965: --- Nice small patch, thank you. This looks good to me. +1 > s3a input stream "normal" fadvise mode to be adaptive > - > > Key: HADOOP-14965 > URL: https://issues.apache.org/jira/browse/HADOOP-14965 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14965-001.patch, HADOOP-14965-002.patch, > HADOOP-14965-003.patch, HADOOP-14965-004.patch > > > HADOOP-14535 added seek optimisation to wasb, but rather than require the > caller to declare sequential vs random, it works out for itself. > # defaults to sequential, lazy seek > # if the caller ever seeks backwards, switches to random IO. > This means that on the use pattern of columnar stores: of go to end of file, > read summary, then go to columns and work forwards, will switch to random IO > after that first seek back (cost: one aborted HTTP connection)/. > Where this should benefit the most is in downstream apps where you are > working with different data sources in the same object store/running of the > same app config, but have different read patterns. I'm seeing exactly this in > some of my spark tests, where it's near impossible to set things up so that > .gz files are read sequentially, but ORC data is read in random IO > I propose the "normal" fadvise => adaptive, sequential==sequential always, > random => random from the outset. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15128) TestViewFileSystem tests are broken in trunk
[ https://issues.apache.org/jira/browse/HADOOP-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296316#comment-16296316 ] Eric Yang edited comment on HADOOP-15128 at 12/20/17 1:15 AM: -- In HADOOP-2381, the discussion thread contains good insights why the original code was written as such. We should throw exception when it is unable to load permission information, and owner, group, permission information should not be null. These safe guards are making sure the file system security is not compromised. The recent change allows other class to extend RawLocalFileSystem base class and circumvent security. HADOOP-10054 must be reverted to prevent security hole. There are other ways to fix ViewFsFileStatus.toString() to make sure the path is initialized properly without modifying FileStatus class. was (Author: eyang): In HADOOP-2381, the discussion thread contains good insights why the original code was written as such. We should throw exception when it is unable to load permission information, and owner, group, permission information should not be null. These safe guards are making sure the file system security is not compromised. The recent change allows other class to extend RawLocalFileSystem base class and circumvent security. HADOOP-10054 must be reverted to prevent security hole. There are other ways to fix ViewFsFileStatus.toString() to make sure the path is initialized properly without modifying RawLocalFileSystem class. > TestViewFileSystem tests are broken in trunk > > > Key: HADOOP-15128 > URL: https://issues.apache.org/jira/browse/HADOOP-15128 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.1.0 >Reporter: Anu Engineer >Assignee: Hanisha Koneru > > The fix in Hadoop-10054 seems to have caused a test failure. Please take a > look. Thanks [~eyang] for reporting this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14630) Contract Tests to verify create, mkdirs and rename under a file is forbidden
[
https://issues.apache.org/jira/browse/HADOOP-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297401#comment-16297401
]
genericqa commented on HADOOP-14630:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
35s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 30s{color}
| {color:red} root generated 1 new + 1232 unchanged - 0 fixed = 1233 total (was
1232) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 5s{color} | {color:orange} root: The patch generated 4 new + 145 unchanged
- 3 fixed = 149 total (was 148) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 59s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 39s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
33s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hadoop-openstack in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
13s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
51s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
| | hadoop.fs.contract.localfs.TestLocalFSContractRename |
| | hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem |
| |
[jira] [Commented] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297389#comment-16297389
]
Karthik Palaniappan commented on HADOOP-15129:
--
[~shahrs87]: It's related to both -- I'll add links to them.
1) This is not related to proxies like 8068 and 12125
2) This is a different proposed fix than 8068 -- instead of adding retries
inside of HDFS, it adds retries down at the IPC layer. This is more in line
with what [~jlowe] suggested in 12125.
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Assignee: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna see the Datanode log: "Namenode...remains
> unresolved" -- referring to the fact that DNS lookup failed.
> {code:java}
> 2017-11-02 16:01:55,115 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Refresh request received for nameservices: null
> 2017-11-02 16:01:55,153 WARN org.apache.hadoop.hdfs.DFSUtilClient: Namenode
> for null remains unresolved for ID null. Check your hdfs-site.xml file to
> ensure namenodes are configured properly.
> 2017-11-02 16:01:55,156 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Starting BPOfferServices for nameservices:
> 2017-11-02 16:01:55,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block pool (Datanode Uuid unassigned) service to
> cluster-32f5-m:8020 starting to offer service
> {code}
> The Datanode then proceeds to use this unresolved address, as it may work if
> the DN is configured to use a proxy. Since I'm not using a proxy, it forever
> prints out this message:
> {code:java}
> 2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:45,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:50,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:55,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:14:00,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> {code}
> Unfortunately, the log doesn't contain the exception that triggered it, but
> the culprit is actually in IPC Client:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L444.
> This line was introduced in https://issues.apache.org/jira/browse/HADOOP-487
> to give a clear error message when somebody mispells an address.
> However, the fix in HADOOP-7472 doesn't apply here, because that code happens
> in Client#getConnection after the Connection is constructed.
> My proposed fix (will attach a patch) is to move this exception out of the
> constructor and into a place that will trigger HADOOP-7472's logic to
> re-resolve addresses. If the DNS failure was temporary, this will allow the
> connection to succeed. If not, the connection will fail after ipc client
> retries (default 10 seconds worth of retries).
> I want to fix this in ipc client rather than just in Datanode startup, as
> this fixes temporary DNS issues for all of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297380#comment-16297380
]
Karthik Palaniappan commented on HADOOP-15129:
--
[~arpitagarwal]: Here are the notes from our investigation.
startDataNode() calls refreshNamenodes():
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L1425
refreshNamenodes has unresolved InetSocketAddress(es):
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolManager.java#L152,
and the unresolved address(es) get passed to a BPOfferService. Note that a
BPOfferService is only created once for a NameServiceId.
BPOS caches that ISA:
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java#L134
And creates a BPServiceActor with that ISA:
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java#L138
When the BPServiceActor tries to get NN info, it hits that exception (and logs
“Problem connecting to server”):
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java#L235
Here is the stacktrace of the exception:
{code:java}
java.net.UnknownHostException: Invalid host name: local host is: (unknown);
destination host is: "non-existent":8020; java.net.UnknownHostException; For
more details see: http://wiki.apache.org/hadoop/UnknownHost
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.retrieveNamespaceInfo(BPServiceActor.java:221)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:263)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:752)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: Invalid host name: local host is:
(unknown); destination host is: "non-existent":8020;
java.net.UnknownHostException; For more details see:
http://wiki.apache.org/hadoop/UnknownHost
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:744)
at org.apache.hadoop.ipc.Client$Connection.(Client.java:446)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1524)
at org.apache.hadoop.ipc.Client.call(Client.java:1375)
at org.apache.hadoop.ipc.Client.call(Client.java:1339)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy15.versionRequest(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.versionRequest(DatanodeProtocolClientSideTranslatorPB.java:274)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.retrieveNamespaceInfo(BPServiceActor.java:216)
... 3 more
{code}
BPServiceActor uses a while(true) loop that runs every 5 seconds. I don’t think
it ever stops (which is a little weird).
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Assignee: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna
[jira] [Updated] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[
https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14707:
Attachment: HADOOP-14707-001.patch
patch 001; adds a test
> AbstractContractDistCpTest to test attr preservation with -p, verify
> blobstores downgrade
> -
>
> Key: HADOOP-14707
> URL: https://issues.apache.org/jira/browse/HADOOP-14707
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/azure, fs/s3, test, tools/distcp
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14707-001.patch
>
>
> It *may* be that trying to use {{distcp -p}} with S3a triggers a stack trace
> {code}
> java.lang.UnsupportedOperationException: S3AFileSystem doesn't support
> getXAttrs
> at org.apache.hadoop.fs.FileSystem.getXAttrs(FileSystem.java:2559)
> at
> org.apache.hadoop.tools.util.DistCpUtils.toCopyListingFileStatus(DistCpUtils.java:322)
>
> {code}
> Add a test to {{AbstractContractDistCpTest}} to verify that this is handled
> better. What is "handle better" here? Either ignore the option or fail with
> "don't do that" text
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14707) AbstractContractDistCpTest to test attr preservation with -p, verify blobstores downgrade
[
https://issues.apache.org/jira/browse/HADOOP-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297331#comment-16297331
]
Steve Loughran commented on HADOOP-14707:
-
+ have filesystems implement StreamCapabilities & declare their support for
xattr and perms
> AbstractContractDistCpTest to test attr preservation with -p, verify
> blobstores downgrade
> -
>
> Key: HADOOP-14707
> URL: https://issues.apache.org/jira/browse/HADOOP-14707
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/azure, fs/s3, test, tools/distcp
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
>
> It *may* be that trying to use {{distcp -p}} with S3a triggers a stack trace
> {code}
> java.lang.UnsupportedOperationException: S3AFileSystem doesn't support
> getXAttrs
> at org.apache.hadoop.fs.FileSystem.getXAttrs(FileSystem.java:2559)
> at
> org.apache.hadoop.tools.util.DistCpUtils.toCopyListingFileStatus(DistCpUtils.java:322)
>
> {code}
> Add a test to {{AbstractContractDistCpTest}} to verify that this is handled
> better. What is "handle better" here? Either ignore the option or fail with
> "don't do that" text
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297246#comment-16297246
]
Arpit Agarwal commented on HADOOP-15129:
Thanks for the heads up Steve.
[~Karthik Palaniappan], do you have the callstack to go with this error message?
{code}
2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Problem connecting to server: cluster-32f5-m:8020
{code}
It's not logged by default but perhaps you captured it while debugging. Also I
added you as a contributor.
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Assignee: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna see the Datanode log: "Namenode...remains
> unresolved" -- referring to the fact that DNS lookup failed.
> {code:java}
> 2017-11-02 16:01:55,115 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Refresh request received for nameservices: null
> 2017-11-02 16:01:55,153 WARN org.apache.hadoop.hdfs.DFSUtilClient: Namenode
> for null remains unresolved for ID null. Check your hdfs-site.xml file to
> ensure namenodes are configured properly.
> 2017-11-02 16:01:55,156 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Starting BPOfferServices for nameservices:
> 2017-11-02 16:01:55,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block pool (Datanode Uuid unassigned) service to
> cluster-32f5-m:8020 starting to offer service
> {code}
> The Datanode then proceeds to use this unresolved address, as it may work if
> the DN is configured to use a proxy. Since I'm not using a proxy, it forever
> prints out this message:
> {code:java}
> 2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:45,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:50,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:55,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:14:00,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> {code}
> Unfortunately, the log doesn't contain the exception that triggered it, but
> the culprit is actually in IPC Client:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L444.
> This line was introduced in https://issues.apache.org/jira/browse/HADOOP-487
> to give a clear error message when somebody mispells an address.
> However, the fix in HADOOP-7472 doesn't apply here, because that code happens
> in Client#getConnection after the Connection is constructed.
> My proposed fix (will attach a patch) is to move this exception out of the
> constructor and into a place that will trigger HADOOP-7472's logic to
> re-resolve addresses. If the DNS failure was temporary, this will allow the
> connection to succeed. If not, the connection will fail after ipc client
> retries (default 10 seconds worth of retries).
> I want to fix this in ipc client rather than just in Datanode startup, as
> this fixes temporary DNS issues for all of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14630) Contract Tests to verify create, mkdirs and rename under a file is forbidden
[ https://issues.apache.org/jira/browse/HADOOP-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14630: Attachment: HADOOP-14630-003.patch Patch 003 Tested: HDFS, s3a ireland, azure ireland, adl ireland, swift rackspace US Fixes to make tests work * Swift supports createNonRecursive(). Not directly related to this patch, but needed for the create tests to complete. * ADL test subclass to expect the AccessControlException and verify its text. * Small patch to one of the Azure tests while looking at an intermittent failure on parallel test runs. No obvious cause, but again, its create() methods The ADL behavior doesn't match the strict policy of "downgrade to a return false", but unless we do want to change its behaviour to be consistent but less informative to callers, its hard to justify changing > Contract Tests to verify create, mkdirs and rename under a file is forbidden > > > Key: HADOOP-14630 > URL: https://issues.apache.org/jira/browse/HADOOP-14630 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure, fs/s3, fs/swift >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-14630-001.patch, HADOOP-14630-002.patch, > HADOOP-14630-003.patch > > > Object stores can get into trouble in ways which an FS would never, do, ways > so obvious we've never done tests for them. We know what the problems are: > test for file and dir creation directly/indirectly under other files > * mkdir(file/file) > * mkdir(file/subdir) > * dir under file/subdir/subdir > * dir/dir2/file, verify dir & dir2 exist > * dir/dir2/dir3, verify dir & dir2 exist > * rename(src, file/dest) > * rename(src, file/dir/dest) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14630) Contract Tests to verify create, mkdirs and rename under a file is forbidden
[ https://issues.apache.org/jira/browse/HADOOP-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14630: Status: Patch Available (was: Open) > Contract Tests to verify create, mkdirs and rename under a file is forbidden > > > Key: HADOOP-14630 > URL: https://issues.apache.org/jira/browse/HADOOP-14630 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure, fs/s3, fs/swift >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-14630-001.patch, HADOOP-14630-002.patch, > HADOOP-14630-003.patch > > > Object stores can get into trouble in ways which an FS would never, do, ways > so obvious we've never done tests for them. We know what the problems are: > test for file and dir creation directly/indirectly under other files > * mkdir(file/file) > * mkdir(file/subdir) > * dir under file/subdir/subdir > * dir/dir2/file, verify dir & dir2 exist > * dir/dir2/dir3, verify dir & dir2 exist > * rename(src, file/dest) > * rename(src, file/dir/dest) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14630) Contract Tests to verify create, mkdirs and rename under a file is forbidden
[ https://issues.apache.org/jira/browse/HADOOP-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14630: Status: Open (was: Patch Available) > Contract Tests to verify create, mkdirs and rename under a file is forbidden > > > Key: HADOOP-14630 > URL: https://issues.apache.org/jira/browse/HADOOP-14630 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure, fs/s3, fs/swift >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-14630-001.patch, HADOOP-14630-002.patch > > > Object stores can get into trouble in ways which an FS would never, do, ways > so obvious we've never done tests for them. We know what the problems are: > test for file and dir creation directly/indirectly under other files > * mkdir(file/file) > * mkdir(file/subdir) > * dir under file/subdir/subdir > * dir/dir2/file, verify dir & dir2 exist > * dir/dir2/dir3, verify dir & dir2 exist > * rename(src, file/dest) > * rename(src, file/dir/dest) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14569) NativeAzureFileSystem, AzureBlobStorageTestAccount to have useful toString() values
[
https://issues.apache.org/jira/browse/HADOOP-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297224#comment-16297224
]
genericqa commented on HADOOP-14569:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
5s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 4s{color} | {color:orange} root: The patch generated 89 new + 45 unchanged
- 0 fixed = 134 total (was 45) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 58s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 59s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
12s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem |
| | hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14569 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880444/HADOOP-14569.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d0126025f738 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk
[jira] [Assigned] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal reassigned HADOOP-15129:
--
Assignee: Karthik Palaniappan
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Assignee: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna see the Datanode log: "Namenode...remains
> unresolved" -- referring to the fact that DNS lookup failed.
> {code:java}
> 2017-11-02 16:01:55,115 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Refresh request received for nameservices: null
> 2017-11-02 16:01:55,153 WARN org.apache.hadoop.hdfs.DFSUtilClient: Namenode
> for null remains unresolved for ID null. Check your hdfs-site.xml file to
> ensure namenodes are configured properly.
> 2017-11-02 16:01:55,156 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Starting BPOfferServices for nameservices:
> 2017-11-02 16:01:55,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block pool (Datanode Uuid unassigned) service to
> cluster-32f5-m:8020 starting to offer service
> {code}
> The Datanode then proceeds to use this unresolved address, as it may work if
> the DN is configured to use a proxy. Since I'm not using a proxy, it forever
> prints out this message:
> {code:java}
> 2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:45,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:50,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:55,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:14:00,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> {code}
> Unfortunately, the log doesn't contain the exception that triggered it, but
> the culprit is actually in IPC Client:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L444.
> This line was introduced in https://issues.apache.org/jira/browse/HADOOP-487
> to give a clear error message when somebody mispells an address.
> However, the fix in HADOOP-7472 doesn't apply here, because that code happens
> in Client#getConnection after the Connection is constructed.
> My proposed fix (will attach a patch) is to move this exception out of the
> constructor and into a place that will trigger HADOOP-7472's logic to
> re-resolve addresses. If the DNS failure was temporary, this will allow the
> connection to succeed. If not, the connection will fail after ipc client
> retries (default 10 seconds worth of retries).
> I want to fix this in ipc client rather than just in Datanode startup, as
> this fixes temporary DNS issues for all of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13764) WASB test runs leak storage containers.
[
https://issues.apache.org/jira/browse/HADOOP-13764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297098#comment-16297098
]
Steve Loughran commented on HADOOP-13764:
-
If you explicitly the test CleanupTestContainers (via {{mvn test
-Dtest=CleanupTestContainers}} then it'll do a list and delete of all these
leaked containers.
> WASB test runs leak storage containers.
> ---
>
> Key: HADOOP-13764
> URL: https://issues.apache.org/jira/browse/HADOOP-13764
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, test
>Reporter: Steve Loughran
>Priority: Minor
>
> It appears that WASB test runs dynamically allocate a container within the
> storage account, using a naming convention of "wasbtests--".
> These containers are not cleaned up automatically, so they remain in the
> storage account indefinitely.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14552) Über-jira: WASB client phase II: performance and testing
[ https://issues.apache.org/jira/browse/HADOOP-14552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-14552. - Resolution: Fixed Assignee: Thomas Marquardt Fix Version/s: 2.9.0 3.0.0 Target Version/s: (was: 3.1.0) Moving everything in this list not in Hadoop 2.9/3.0 to HADOOP-15132; this JIRA now summarizes everything new w.r.t WASB integration. > Über-jira: WASB client phase II: performance and testing > > > Key: HADOOP-14552 > URL: https://issues.apache.org/jira/browse/HADOOP-14552 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Thomas Marquardt > Fix For: 3.0.0, 2.9.0 > > > Uber-JIRA for wasb:// phase II: the things we want for Hadoop 2.9. > I think the focus is on performance, but looking at the tests there are some > things we need there first: parallel test execution, move over to IT tests, > other tuning. > All patches must come with declarations of which azure endpoint they were > tested against. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13764) WASB test runs leak storage containers.
[ https://issues.apache.org/jira/browse/HADOOP-13764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13764: Priority: Minor (was: Major) Component/s: test > WASB test runs leak storage containers. > --- > > Key: HADOOP-13764 > URL: https://issues.apache.org/jira/browse/HADOOP-13764 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Reporter: Steve Loughran >Priority: Minor > > It appears that WASB test runs dynamically allocate a container within the > storage account, using a naming convention of "wasbtests--". > These containers are not cleaned up automatically, so they remain in the > storage account indefinitely. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13764) WASB test runs leak storage containers.
[ https://issues.apache.org/jira/browse/HADOOP-13764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13764: Issue Type: Sub-task (was: Bug) Parent: HADOOP-15132 > WASB test runs leak storage containers. > --- > > Key: HADOOP-13764 > URL: https://issues.apache.org/jira/browse/HADOOP-13764 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Reporter: Steve Loughran > > It appears that WASB test runs dynamically allocate a container within the > storage account, using a naming convention of "wasbtests--". > These containers are not cleaned up automatically, so they remain in the > storage account indefinitely. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14582) WASB exception handling to translate specific failures into specific exceptions
[
https://issues.apache.org/jira/browse/HADOOP-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14582:
Parent Issue: HADOOP-15132 (was: HADOOP-14552)
> WASB exception handling to translate specific failures into specific
> exceptions
> ---
>
> Key: HADOOP-14582
> URL: https://issues.apache.org/jira/browse/HADOOP-14582
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>
> Mucht of the exception handling logic in WASB is a catch-and-wrap-as-IOE
> handler.
> {code}
> } catch (Exception e) {
> // Re-throw the exception as an Azure storage exception.
> throw new AzureException(e);
> }
> {code}
> This works, but
> # it doesn't offer much in the way of diagnostics
> # There's no way to make sense of the failure
> # It catches and wraps IOEs, which don't need wrapping
> # It can double wrap IOEs around storage exceptions. Example
> {{PageBlobInputStream}} constructor wraps StorageException with IOE; if
> raised in {{AzureNativeFileSystemStore.retrieve()}} it will be caught and
> wrapped again.
> Proposed: something akin to where S3A's translateException is going:
> * take an incoming StorageException and based on its errorcode, choose
> whether or not to wrap in a specific java exception (FNFE, access denied,
> ...).
> * {{AzureException}} to make it easy to get at the details
> * Include operation & path in error text
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14569) NativeAzureFileSystem, AzureBlobStorageTestAccount to have useful toString() values
[
https://issues.apache.org/jira/browse/HADOOP-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14569:
Status: Patch Available (was: In Progress)
> NativeAzureFileSystem, AzureBlobStorageTestAccount to have useful toString()
> values
> ---
>
> Key: HADOOP-14569
> URL: https://issues.apache.org/jira/browse/HADOOP-14569
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14569.001.patch, formatted_output
>
>
> {{NativeAzureFileSystem.toString()}}, and
> {{AzureBlobStorageTestAccount.toString()}} should return data meaningful in
> logging & test runs
> * account name
> * container name/status
> * ideally, FS instrumentation statistics
> * + not to NPE if invoked before calling FileSystem.initialize(), or after
> being closed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14569) NativeAzureFileSystem, AzureBlobStorageTestAccount to have useful toString() values
[
https://issues.apache.org/jira/browse/HADOOP-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HADOOP-14569:
---
Assignee: Amit Singh (was: Steve Loughran)
> NativeAzureFileSystem, AzureBlobStorageTestAccount to have useful toString()
> values
> ---
>
> Key: HADOOP-14569
> URL: https://issues.apache.org/jira/browse/HADOOP-14569
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Amit Singh
>Priority: Minor
> Attachments: HADOOP-14569.001.patch, formatted_output
>
>
> {{NativeAzureFileSystem.toString()}}, and
> {{AzureBlobStorageTestAccount.toString()}} should return data meaningful in
> logging & test runs
> * account name
> * container name/status
> * ideally, FS instrumentation statistics
> * + not to NPE if invoked before calling FileSystem.initialize(), or after
> being closed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14569) NativeAzureFileSystem, AzureBlobStorageTestAccount to have useful toString() values
[
https://issues.apache.org/jira/browse/HADOOP-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HADOOP-14569:
---
Assignee: Steve Loughran (was: Amit Singh)
> NativeAzureFileSystem, AzureBlobStorageTestAccount to have useful toString()
> values
> ---
>
> Key: HADOOP-14569
> URL: https://issues.apache.org/jira/browse/HADOOP-14569
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14569.001.patch, formatted_output
>
>
> {{NativeAzureFileSystem.toString()}}, and
> {{AzureBlobStorageTestAccount.toString()}} should return data meaningful in
> logging & test runs
> * account name
> * container name/status
> * ideally, FS instrumentation statistics
> * + not to NPE if invoked before calling FileSystem.initialize(), or after
> being closed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14569) NativeAzureFileSystem, AzureBlobStorageTestAccount to have useful toString() values
[
https://issues.apache.org/jira/browse/HADOOP-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297069#comment-16297069
]
Steve Loughran commented on HADOOP-14569:
-
sorry, I'd missed this!
thanks for this. I'm about to fiddle with the JIRA settings to have yetus do a
test run. I expect it will reject the one line {{if(container == null) return
null;}} clauses
Better to have a multiline condition
{code}
if (container == null) {
return none;
}
{code}
Otherwise, code LGTM
> NativeAzureFileSystem, AzureBlobStorageTestAccount to have useful toString()
> values
> ---
>
> Key: HADOOP-14569
> URL: https://issues.apache.org/jira/browse/HADOOP-14569
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Amit Singh
>Priority: Minor
> Attachments: HADOOP-14569.001.patch, formatted_output
>
>
> {{NativeAzureFileSystem.toString()}}, and
> {{AzureBlobStorageTestAccount.toString()}} should return data meaningful in
> logging & test runs
> * account name
> * container name/status
> * ideally, FS instrumentation statistics
> * + not to NPE if invoked before calling FileSystem.initialize(), or after
> being closed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14700) NativeAzureFileSystem.open() ignores blob container name
[
https://issues.apache.org/jira/browse/HADOOP-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14700:
Parent Issue: HADOOP-15132 (was: HADOOP-14552)
> NativeAzureFileSystem.open() ignores blob container name
>
>
> Key: HADOOP-14700
> URL: https://issues.apache.org/jira/browse/HADOOP-14700
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-beta1, 3.0.0-alpha4
>Reporter: Cheng Lian
>
> {{NativeAzureFileSystem}} instances are associated with the blob container
> used to initialize the file system. Assuming that a file system instance
> {{fs}} is associated with a container {{A}}, when trying to access a blob
> inside another container {{B}}, {{fs}} still tries to find the blob inside
> container {{A}}. If there happens to be two blobs with the same name inside
> both containers, the user may get a wrong result because {{fs}} reads the
> contents from the blob inside container {{A}} instead of container {{B}}.
> You may reproduce it by running the following self-contained Scala script
> using [Ammonite|http://ammonite.io/]:
> {code}
> #!/usr/bin/env amm --no-remote-logging
> import $ivy.`com.jsuereth::scala-arm:2.0`
> import $ivy.`com.microsoft.azure:azure-storage:5.2.0`
> import $ivy.`org.apache.hadoop:hadoop-azure:3.0.0-alpha4`
> import $ivy.`org.apache.hadoop:hadoop-common:3.0.0-alpha4`
> import $ivy.`org.scalatest::scalatest:3.0.3`
> import java.io.{BufferedReader, InputStreamReader}
> import java.net.URI
> import java.time.{Duration, Instant}
> import java.util.{Date, EnumSet}
> import com.microsoft.azure.storage.{CloudStorageAccount,
> StorageCredentialsAccountAndKey}
> import com.microsoft.azure.storage.blob.{SharedAccessBlobPermissions,
> SharedAccessBlobPolicy}
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> import org.apache.hadoop.fs.azure.{AzureException, NativeAzureFileSystem}
> import org.scalatest.Assertions._
> import resource._
> // Utility implicit conversion for auto resource management.
> implicit def `Closable->Resource`[T <: { def close() }]: Resource[T] = new
> Resource[T] {
> override def close(closable: T): Unit = closable.close()
> }
> // Credentials information
> val ACCOUNT = "** REDACTED **"
> val ACCESS_KEY = "** REDACTED **"
> // We'll create two different containers, both contain a blob named
> "test-blob" but with different
> // contents.
> val CONTAINER_A = "container-a"
> val CONTAINER_B = "container-b"
> val TEST_BLOB = "test-blob"
> val blobClient = {
> val credentials = new StorageCredentialsAccountAndKey(ACCOUNT, ACCESS_KEY)
> val account = new CloudStorageAccount(credentials, /* useHttps */ true)
> account.createCloudBlobClient()
> }
> // Generates a read-only SAS key restricted within "container-a".
> val sasKeyForContainerA = {
> val since = Instant.now() minus Duration.ofMinutes(10)
> val duration = Duration.ofHours(1)
> val policy = new SharedAccessBlobPolicy()
> policy.setSharedAccessStartTime(Date.from(since))
> policy.setSharedAccessExpiryTime(Date.from(since plus duration))
> policy.setPermissions(EnumSet.of(
> SharedAccessBlobPermissions.READ,
> SharedAccessBlobPermissions.LIST
> ))
> blobClient
> .getContainerReference(CONTAINER_A)
> .generateSharedAccessSignature(policy, null)
> }
> // Sets up testing containers and blobs using the Azure storage SDK:
> //
> // container-a/test-blob => "foo"
> // container-b/test-blob => "bar"
> {
> val containerARef = blobClient.getContainerReference(CONTAINER_A)
> val containerBRef = blobClient.getContainerReference(CONTAINER_B)
> containerARef.createIfNotExists()
> containerARef.getBlockBlobReference(TEST_BLOB).uploadText("foo")
> containerBRef.createIfNotExists()
> containerBRef.getBlockBlobReference(TEST_BLOB).uploadText("bar")
> }
> val pathA = new
> Path(s"wasbs://$CONTAINER_A@$ACCOUNT.blob.core.windows.net/$TEST_BLOB")
> val pathB = new
> Path(s"wasbs://$CONTAINER_B@$ACCOUNT.blob.core.windows.net/$TEST_BLOB")
> for {
> // Creates a file system associated with "container-a".
> fs <- managed {
> val conf = new Configuration
> conf.set("fs.wasbs.impl", classOf[NativeAzureFileSystem].getName)
> conf.set(s"fs.azure.sas.$CONTAINER_A.$ACCOUNT.blob.core.windows.net",
> sasKeyForContainerA)
> pathA.getFileSystem(conf)
> }
> // Opens a reader pointing to "container-a/test-blob". We expect to get the
> string "foo" written
> // to this blob previously.
> readerA <- managed(new BufferedReader(new InputStreamReader(fs open pathA)))
> // Opens a reader pointing to "container-b/test-blob". We expect to get an
> exception since the SAS
> // key used to create the `FileSystem` instance is restricted to
>
[jira] [Updated] (HADOOP-14569) NativeAzureFileSystem, AzureBlobStorageTestAccount to have useful toString() values
[
https://issues.apache.org/jira/browse/HADOOP-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14569:
Parent Issue: HADOOP-15132 (was: HADOOP-14552)
> NativeAzureFileSystem, AzureBlobStorageTestAccount to have useful toString()
> values
> ---
>
> Key: HADOOP-14569
> URL: https://issues.apache.org/jira/browse/HADOOP-14569
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Amit Singh
>Priority: Minor
> Attachments: HADOOP-14569.001.patch, formatted_output
>
>
> {{NativeAzureFileSystem.toString()}}, and
> {{AzureBlobStorageTestAccount.toString()}} should return data meaningful in
> logging & test runs
> * account name
> * container name/status
> * ideally, FS instrumentation statistics
> * + not to NPE if invoked before calling FileSystem.initialize(), or after
> being closed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14612) Reduce memory copy in BlobOutputStreamInternal::dispatchWrite
[
https://issues.apache.org/jira/browse/HADOOP-14612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14612:
Parent Issue: HADOOP-15132 (was: HADOOP-14552)
> Reduce memory copy in BlobOutputStreamInternal::dispatchWrite
> -
>
> Key: HADOOP-14612
> URL: https://issues.apache.org/jira/browse/HADOOP-14612
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Rajesh Balamohan
>Priority: Minor
>
> Currently in {{BlobOutputStreamInternal::dispatchWrite}}, buffer is copied
> internally for every write. During large uploads this can be around 4 MB.
> This can be avoided if there is internal class which extends
> ByteArrayOutputStream with additional method "ByteArrayInputStream
> getInputStream()".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14613) TestNativeAzureFileSystemOperationsMocked creating files in /tmp
[
https://issues.apache.org/jira/browse/HADOOP-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14613:
Parent Issue: HADOOP-15132 (was: HADOOP-14552)
> TestNativeAzureFileSystemOperationsMocked creating files in /tmp
>
>
> Key: HADOOP-14613
> URL: https://issues.apache.org/jira/browse/HADOOP-14613
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, test
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
>
> {{TestNativeAzureFileSystemOperationsMocked}} is creating temp files in
> "/tmp", on account of it having a hard-code path for temp data of {{
> "/tmp/TestNativeAzureFileSystemOperationsMocked"}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14748) Wasb input streams to implement CanUnbuffer
[
https://issues.apache.org/jira/browse/HADOOP-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14748:
Parent Issue: HADOOP-15132 (was: HADOOP-14552)
> Wasb input streams to implement CanUnbuffer
> ---
>
> Key: HADOOP-14748
> URL: https://issues.apache.org/jira/browse/HADOOP-14748
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
>
> HBase relies on FileSystems implementing {{CanUnbuffer.unbuffer()}} to force
> input streams to free up remote connections (HBASE-9393Link). This works for
> HDFS, but not elsewhere.
> WASB {{BlockBlobInputStream}} can implement this by closing the stream in
> {{closeBlobInputStream}}, so it will be re-opened elsewhere.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14755) WASB to implement listFiles(Path f, boolean recursive) through flat list
[
https://issues.apache.org/jira/browse/HADOOP-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14755:
Parent Issue: HADOOP-15132 (was: HADOOP-14552)
> WASB to implement listFiles(Path f, boolean recursive) through flat list
> -
>
> Key: HADOOP-14755
> URL: https://issues.apache.org/jira/browse/HADOOP-14755
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> WASB doesn't override {{FileSystem.listFiles(Path f, boolean recursive)}}, so
> picks up the base treewalk implementation. As the blobstore does implement
> deep listing itself, it should be "straightforward" to implement this
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14755) WASB to implement listFiles(Path f, boolean recursive) through flat list
[
https://issues.apache.org/jira/browse/HADOOP-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297063#comment-16297063
]
Steve Loughran commented on HADOOP-14755:
-
Wasb has a flat list option for delete & rename, so it can be exposed here too
> WASB to implement listFiles(Path f, boolean recursive) through flat list
> -
>
> Key: HADOOP-14755
> URL: https://issues.apache.org/jira/browse/HADOOP-14755
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> WASB doesn't override {{FileSystem.listFiles(Path f, boolean recursive)}}, so
> picks up the base treewalk implementation. As the blobstore does implement
> deep listing itself, it should be "straightforward" to implement this
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14767) WASB to implement copyFromLocalFile()
[
https://issues.apache.org/jira/browse/HADOOP-14767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14767:
Affects Version/s: (was: 2.8.1)
2.9.0
Priority: Minor (was: Major)
> WASB to implement copyFromLocalFile()
> -
>
> Key: HADOOP-14767
> URL: https://issues.apache.org/jira/browse/HADOOP-14767
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
>
> WASB just uses the default FS copyFromLocalFile. If HADOOP-14766 adds an
> object-store-friendly upload command, wasb would benefit the most if it had a
> {{copyFromLocalFile()}} command tuned to make the most of the API.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14767) WASB to implement copyFromLocalFile()
[
https://issues.apache.org/jira/browse/HADOOP-14767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14767:
Parent Issue: HADOOP-15132 (was: HADOOP-14552)
> WASB to implement copyFromLocalFile()
> -
>
> Key: HADOOP-14767
> URL: https://issues.apache.org/jira/browse/HADOOP-14767
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>
> WASB just uses the default FS copyFromLocalFile. If HADOOP-14766 adds an
> object-store-friendly upload command, wasb would benefit the most if it had a
> {{copyFromLocalFile()}} command tuned to make the most of the API.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15086) NativeAzureFileSystem file rename is not atomic
[ https://issues.apache.org/jira/browse/HADOOP-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15086: Summary: NativeAzureFileSystem file rename is not atomic (was: NativeAzureFileSystem.rename is not atomic) > NativeAzureFileSystem file rename is not atomic > --- > > Key: HADOOP-15086 > URL: https://issues.apache.org/jira/browse/HADOOP-15086 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.3 >Reporter: Shixiong Zhu > Attachments: RenameReproducer.java > > > When multiple threads rename files to the same target path, more than 1 > threads can succeed. It's because check and copy file in `rename` is not > atomic. > I would expect it's atomic just like HDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15086) NativeAzureFileSystem.rename is not atomic
[ https://issues.apache.org/jira/browse/HADOOP-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15086: Parent Issue: HADOOP-15132 (was: HADOOP-14552) > NativeAzureFileSystem.rename is not atomic > -- > > Key: HADOOP-15086 > URL: https://issues.apache.org/jira/browse/HADOOP-15086 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.3 >Reporter: Shixiong Zhu > Attachments: RenameReproducer.java > > > When multiple threads rename files to the same target path, more than 1 > threads can succeed. It's because check and copy file in `rename` is not > atomic. > I would expect it's atomic just like HDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15082) add AbstractContractRootDirectoryTest test for mkdir / ; wasb to implement the test
[
https://issues.apache.org/jira/browse/HADOOP-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15082:
Parent Issue: HADOOP-15132 (was: HADOOP-14552)
> add AbstractContractRootDirectoryTest test for mkdir / ; wasb to implement
> the test
> ---
>
> Key: HADOOP-15082
> URL: https://issues.apache.org/jira/browse/HADOOP-15082
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure, test
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-15082-001.patch, HADOOP-15082-002.patch
>
>
> I managed to get a stack trace on an older version of WASB with some coding
> doing a mkdir(new Path("/"))some of the ranger parentage checks didn't
> handle that specific case.
> # Add a new root Fs contract test for this operation
> # Have WASB implement the test suite as an integration test.
> # if the test fails shows a problem fix
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15044) Wasb getFileBlockLocations() returns too many locations.
[
https://issues.apache.org/jira/browse/HADOOP-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15044:
Parent Issue: HADOOP-15132 (was: HADOOP-14552)
> Wasb getFileBlockLocations() returns too many locations.
>
>
> Key: HADOOP-15044
> URL: https://issues.apache.org/jira/browse/HADOOP-15044
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Steve Loughran
>
> The wasb mimicking of {{getFileBlockLocations()}} uses the length of the file
> as the number to use to calculate the # of blocks to create (i.e.
> file.length/blocksize), when it should be just the range of the request.
> As a result, you always get the number of blocks in the total file, not the
> number spanning the range of (start, len). If this is less (i.e start > 0 or
> len < file.length), you end up with some 0-byte-range blocks at the end
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15044) Wasb getFileBlockLocations() returns too many locations.
[
https://issues.apache.org/jira/browse/HADOOP-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HADOOP-15044:
---
Assignee: Steve Loughran
> Wasb getFileBlockLocations() returns too many locations.
>
>
> Key: HADOOP-15044
> URL: https://issues.apache.org/jira/browse/HADOOP-15044
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> The wasb mimicking of {{getFileBlockLocations()}} uses the length of the file
> as the number to use to calculate the # of blocks to create (i.e.
> file.length/blocksize), when it should be just the range of the request.
> As a result, you always get the number of blocks in the total file, not the
> number spanning the range of (start, len). If this is less (i.e start > 0 or
> len < file.length), you end up with some 0-byte-range blocks at the end
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14778) AzureNativeFileSystemStore.getDirectorySet() to use getTrimmedStrings to get directory listings
[
https://issues.apache.org/jira/browse/HADOOP-14778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14778:
Parent Issue: HADOOP-15132 (was: HADOOP-14552)
> AzureNativeFileSystemStore.getDirectorySet() to use getTrimmedStrings to get
> directory listings
> ---
>
> Key: HADOOP-14778
> URL: https://issues.apache.org/jira/browse/HADOOP-14778
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Priority: Minor
>
> {{AzureNativeFileSystemStore.getDirectorySet()}} should switch to
> {{Configuration.getTrimmedStrings()}} so the list of directories get
> whitespace and line endings removed.
> As it stands, all paths need to go one the same line, without whitespace
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14941) Azure TestClientThrottlingAnalyzer brittle
[
https://issues.apache.org/jira/browse/HADOOP-14941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14941:
Parent Issue: HADOOP-15132 (was: HADOOP-14552)
> Azure TestClientThrottlingAnalyzer brittle
> --
>
> Key: HADOOP-14941
> URL: https://issues.apache.org/jira/browse/HADOOP-14941
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, test
>Affects Versions: 3.1.0
> Environment: running hadoop-azure tests from remote client.
>Reporter: Steve Loughran
>Priority: Minor
>
> {{TestClientThrottlingAnalyzer}} fails intermittently on long-haul tests
> {code}
>
> TestClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49
> The actual value 10 is not within the expected range: [5.60, 8.40].
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11715) azureFs::getFileStatus doesn't check the file system scheme and thus could throw a misleading exception.
[
https://issues.apache.org/jira/browse/HADOOP-11715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297052#comment-16297052
]
genericqa commented on HADOOP-11715:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-11715 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-11715 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12729101/HADOOP-11715.3.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13856/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> azureFs::getFileStatus doesn't check the file system scheme and thus could
> throw a misleading exception.
> -
>
> Key: HADOOP-11715
> URL: https://issues.apache.org/jira/browse/HADOOP-11715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure
>Affects Versions: 2.7.0
>Reporter: Brandon Li
>Assignee: nijel
> Labels: BB2015-05-TBR
> Attachments: HADOOP-11715.1.patch, HADOOP-11715.2.patch,
> HADOOP-11715.3.patch
>
>
> azureFs::getFileStatus doesn't check the file system scheme and thus could
> throw a misleading exception.
> For example, it complains filenotfound instead of wrong-fs for an hdfs path:
> Caused by: java.io.FileNotFoundException:
> hdfs://headnode0:9000/hive/scratch/hadoopqa/a7d34a22-57eb-4678-84b4-43d84027d45f/hive_2015-03-02_23-13-04_713_5722627238053417441-1/hadoopqa/_tez_scratch_dir/_tez_scratch_dir/split_Map_1/job.split:
> No such file or directory.
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1625)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14906) ITestAzureConcurrentOutOfBandIo failing: The MD5 value specified in the request did not match with the MD5 value calculated by the server
[
https://issues.apache.org/jira/browse/HADOOP-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14906:
Parent Issue: HADOOP-15132 (was: HADOOP-14552)
> ITestAzureConcurrentOutOfBandIo failing: The MD5 value specified in the
> request did not match with the MD5 value calculated by the server
> -
>
> Key: HADOOP-14906
> URL: https://issues.apache.org/jira/browse/HADOOP-14906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.1.0
> Environment: UK BT ASDL connection, 1.8.0_121-b13, azure storage
> ireland
>Reporter: Steve Loughran
>
> {{ITestAzureConcurrentOutOfBandIo}} is consistently raising an IOE with the
> text "The MD5 value specified in the request did not match with the MD5 value
> calculated by the server"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11715) azureFs::getFileStatus doesn't check the file system scheme and thus could throw a misleading exception.
[ https://issues.apache.org/jira/browse/HADOOP-11715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-11715: Parent Issue: HADOOP-15132 (was: HADOOP-14552) > azureFs::getFileStatus doesn't check the file system scheme and thus could > throw a misleading exception. > - > > Key: HADOOP-11715 > URL: https://issues.apache.org/jira/browse/HADOOP-11715 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure >Affects Versions: 2.7.0 >Reporter: Brandon Li >Assignee: nijel > Labels: BB2015-05-TBR > Attachments: HADOOP-11715.1.patch, HADOOP-11715.2.patch, > HADOOP-11715.3.patch > > > azureFs::getFileStatus doesn't check the file system scheme and thus could > throw a misleading exception. > For example, it complains filenotfound instead of wrong-fs for an hdfs path: > Caused by: java.io.FileNotFoundException: > hdfs://headnode0:9000/hive/scratch/hadoopqa/a7d34a22-57eb-4678-84b4-43d84027d45f/hive_2015-03-02_23-13-04_713_5722627238053417441-1/hadoopqa/_tez_scratch_dir/_tez_scratch_dir/split_Map_1/job.split: > No such file or directory. > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:1625) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14584) WASB to support high-performance commit protocol
[ https://issues.apache.org/jira/browse/HADOOP-14584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14584: Parent Issue: HADOOP-15132 (was: HADOOP-14552) > WASB to support high-performance commit protocol > > > Key: HADOOP-14584 > URL: https://issues.apache.org/jira/browse/HADOOP-14584 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Steve Loughran > > Once MAPREDUCE-6823 allows FileOutputFormat to take alternate committers, and > HADOOP-13786 provides the first implementation and tests of a blobstore > specific committer, WASB could do its own. The same strategy: upload > uncommitted blobs and coalesce at the end should work; the same marshalling > of lists of etags *probably* work the same, though there will inevitably be > some subtle differences. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14572) General improvements to Azure test cases
[ https://issues.apache.org/jira/browse/HADOOP-14572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14572: Parent Issue: HADOOP-15132 (was: HADOOP-14552) > General improvements to Azure test cases > > > Key: HADOOP-14572 > URL: https://issues.apache.org/jira/browse/HADOOP-14572 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > > JIRA to track list of things to tweak in Azure tests > including > * replace all uses of System.out with logging > * assertions to have error text > * tests to have timeouts (can be done in base class) > * move from hard coded text in test scans of exception text to string > constants shared across production and test code -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14554) TestFileSystemOperationExceptionMessage to not rerun all of NativeAzureFileSystemBaseTest
[
https://issues.apache.org/jira/browse/HADOOP-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-14554.
-
Resolution: Duplicate
> TestFileSystemOperationExceptionMessage to not rerun all of
> NativeAzureFileSystemBaseTest
> -
>
> Key: HADOOP-14554
> URL: https://issues.apache.org/jira/browse/HADOOP-14554
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>
> the Wasb test {{TestFileSystemOperationExceptionMessage}} subclasses
> {{NativeAzureFileSystemBaseTest}} to test a new situation. As a result, when
> you run this test, it reruns all those base test cases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14591) wasb to support getStorageStatistics
[ https://issues.apache.org/jira/browse/HADOOP-14591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14591: Parent Issue: HADOOP-15132 (was: HADOOP-14552) > wasb to support getStorageStatistics > > > Key: HADOOP-14591 > URL: https://issues.apache.org/jira/browse/HADOOP-14591 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.8.1 >Reporter: Steve Loughran > > Wasb can support the storage statstics of HADOOP-13065, so any execution > framework gathering the statistics can include them, and tests can log them -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13733) Support WASB connections through an HTTP proxy server.
[ https://issues.apache.org/jira/browse/HADOOP-13733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13733: Parent Issue: HADOOP-15132 (was: HADOOP-14552) > Support WASB connections through an HTTP proxy server. > -- > > Key: HADOOP-13733 > URL: https://issues.apache.org/jira/browse/HADOOP-13733 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Chris Nauroth > > WASB currently does not support use of an HTTP proxy server to connect to the > Azure Storage back-end. The Azure Storage SDK does support use of a proxy, > so we can enhance WASB to read proxy settings from configuration and pass > them along in the Azure Storage SDK calls. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14988) WASB: Expose WASB status metrics as counters in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-14988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14988:
Issue Type: Sub-task (was: Bug)
Parent: HADOOP-15132
> WASB: Expose WASB status metrics as counters in Hadoop
> --
>
> Key: HADOOP-14988
> URL: https://issues.apache.org/jira/browse/HADOOP-14988
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Rajesh Balamohan
>Priority: Minor
>
> It would be good to expose WASB status metrics (e.g 503) as Hadoop counters.
> Here is an example from a spark job, where it ends up spending large amount
> of time in retries. Adding hadoop counters would help in analyzing and tuning
> long running tasks.
> {noformat}
> 2017-10-23 23:07:20,876 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: SendingRequest:
> threadId=99, requestType=read , isFirstRequest=false, sleepDuration=0
> 2017-10-23 23:07:20,877 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: ResponseReceived:
> threadId=99, Status=503, Elapsed(ms)=1, ETAG=null, contentLength=198,
> requestMethod=GET
> 2017-10-23 23:07:21,877 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: SendingRequest:
> threadId=99, requestType=read , isFirstRequest=false, sleepDuration=0
> 2017-10-23 23:07:21,879 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: ResponseReceived:
> threadId=99, Status=503, Elapsed(ms)=2, ETAG=null, contentLength=198,
> requestMethod=GET
> 2017-10-23 23:07:24,070 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: SendingRequest:
> threadId=99, requestType=read , isFirstRequest=false, sleepDuration=0
> 2017-10-23 23:07:24,073 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: q:: ResponseReceived: threadId=99, Status=503,
> Elapsed(ms)=3, ETAG=null, contentLength=198, requestMethod=GET
> 2017-10-23 23:07:27,917 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: SendingRequest:
> threadId=99, requestType=read , isFirstRequest=false, sleepDuration=0
> 2017-10-23 23:07:27,920 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: ResponseReceived:
> threadId=99, Status=503, Elapsed(ms)=2, ETAG=null, contentLength=198,
> requestMethod=GET
> 2017-10-23 23:07:36,879 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: SendingRequest:
> threadId=99, requestType=read , isFirstRequest=false, sleepDuration=0
> 2017-10-23 23:07:36,881 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: ResponseReceived:
> threadId=99, Status=503, Elapsed(ms)=1, ETAG=null, contentLength=198,
> requestMethod=GET
> 2017-10-23 23:07:54,786 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: SendingRequest:
> threadId=99, requestType=read , isFirstRequest=false, sleepDuration=0
> 2017-10-23 23:07:54,789 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: ResponseReceived:
> threadId=99, Status=503, Elapsed(ms)=3, ETAG=null, contentLength=198,
> requestMethod=GET
> 2017-10-23 23:08:24,790 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: SendingRequest:
> threadId=99, requestType=read , isFirstRequest=false, sleepDuration=0
> 2017-10-23 23:08:24,794 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: ResponseReceived:
> threadId=99, Status=503, Elapsed(ms)=4, ETAG=null, contentLength=198,
> requestMethod=GET
> 2017-10-23 23:08:54,794 DEBUG [Executor task launch worker for task 2463]
> azure.SelfThrottlingIntercept: SelfThrottlingIntercept:: SendingRequest:
> threadId=99, requestType=read , isFirstRequest=false, sleepDuration=0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13783) Improve efficiency of WASB over page blobs
[ https://issues.apache.org/jira/browse/HADOOP-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13783: Parent Issue: HADOOP-15132 (was: HADOOP-14552) > Improve efficiency of WASB over page blobs > -- > > Key: HADOOP-13783 > URL: https://issues.apache.org/jira/browse/HADOOP-13783 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: NITIN VERMA >Assignee: NITIN VERMA > > 1)Add telemetry to WASB driver. WASB driver is lack of any log or > telemetry which makes trouble shoot very difficult. For example, we don’t > know where is high e2e latency between HBase and Azure storage came from when > Azure storage server latency was very low. Also we don’t know why WASB can > only do 166 IOPs which is way below azure storage 500 IOPs. And we had > several incidents before related to storage latency, because of lacking logs, > we couldn’t find the ownership of the incident quickly. > 2)Resolving the hot spotting issue when WASB driver partition the azure > page blobs by changing the key. Current key design is causing the hot > spotting on azure storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11196) Implement isFileClosed in WASB to make Flume happy
[ https://issues.apache.org/jira/browse/HADOOP-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297047#comment-16297047 ] Steve Loughran commented on HADOOP-11196: - I think Flume shouldn't be expecting every FS to be HDFS, or at least not complaining so much. > Implement isFileClosed in WASB to make Flume happy > -- > > Key: HADOOP-11196 > URL: https://issues.apache.org/jira/browse/HADOOP-11196 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.5.1 >Reporter: Mostafa Elhemali >Priority: Minor > > Flume uses reflection to call isFileClosed() which is a method implemented by > DFS but not part of the FileSystem abstract base class. It still works if the > function is not implemented, but issues a lot of warnings and is distracting. > It would help make it work smoother if we added the same method in the > NativeAzureFileSystem class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13134: Parent Issue: HADOOP-15132 (was: HADOOP-14552) > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Dushyanth > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297046#comment-16297046 ] Steve Loughran commented on HADOOP-13134: - [~zichensun] thanks for that little hint; does make me thing flat listing is the solution. Probably a performance booster too. might be worth changing to make the default > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Dushyanth > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11196) Implement isFileClosed in WASB to make Flume happy
[ https://issues.apache.org/jira/browse/HADOOP-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-11196: Parent Issue: HADOOP-15132 (was: HADOOP-14552) > Implement isFileClosed in WASB to make Flume happy > -- > > Key: HADOOP-11196 > URL: https://issues.apache.org/jira/browse/HADOOP-11196 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.5.1 >Reporter: Mostafa Elhemali >Priority: Minor > > Flume uses reflection to call isFileClosed() which is a method implemented by > DFS but not part of the FileSystem abstract base class. It still works if the > function is not implemented, but issues a lot of warnings and is distracting. > It would help make it work smoother if we added the same method in the > NativeAzureFileSystem class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13754) Hadoop-Azure Update WASB URI format to support SAS token in it.
[ https://issues.apache.org/jira/browse/HADOOP-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13754: Parent Issue: HADOOP-15132 (was: HADOOP-14552) > Hadoop-Azure Update WASB URI format to support SAS token in it. > --- > > Key: HADOOP-13754 > URL: https://issues.apache.org/jira/browse/HADOOP-13754 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sumit Dubey >Assignee: Sumit Dubey > Attachments: HADOOP-13754-branch-2.7.3.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > Currently Azure WASB adapter code supports wasb url in this format > wasb://[containername@]youraccount.blob.core.windows.net/testDir with the > credentials retrieved from configuration and scoped to a container. > With this change we want > 1) change the url to contain file level sas token in the url > wasb://[containername[:]]@youraccount.blob.core.windows.net/testDir > 2) Scope access to a blob/file level. > 3) Tests to test the new url format -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14610) Block compaction for WASB (Block Blobs Instead of Page Blobs)
[ https://issues.apache.org/jira/browse/HADOOP-14610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14610: Parent Issue: HADOOP-15132 (was: HADOOP-14552) > Block compaction for WASB (Block Blobs Instead of Page Blobs) > - > > Key: HADOOP-14610 > URL: https://issues.apache.org/jira/browse/HADOOP-14610 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15132) Über-jira: WASB client phase III: roll-up for Hadoop 3.1
Steve Loughran created HADOOP-15132: --- Summary: Über-jira: WASB client phase III: roll-up for Hadoop 3.1 Key: HADOOP-15132 URL: https://issues.apache.org/jira/browse/HADOOP-15132 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Affects Versions: 3.0.0 Reporter: Steve Loughran Everything for the WASB connector for Hadoop 3.1 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14610) Block compaction for WASB (Block Blobs Instead of Page Blobs)
[ https://issues.apache.org/jira/browse/HADOOP-14610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14610: Summary: Block compaction for WASB (Block Blobs Instead of Page Blobs) (was: Block compaction for WASB (Block Blobs Instead of Page Plobs)) > Block compaction for WASB (Block Blobs Instead of Page Blobs) > - > > Key: HADOOP-14610 > URL: https://issues.apache.org/jira/browse/HADOOP-14610 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[
https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297031#comment-16297031
]
ASF GitHub Bot commented on HADOOP-15033:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/291#discussion_r157801677
--- Diff:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
---
@@ -89,6 +89,22 @@ public static boolean isJava7OrAbove() {
return true;
}
+ private static final boolean IS_JAVA9_OR_ABOVE;
+ static {
+int major = 0;
+try {
+ // 1.8.0 -> 1, 10-ea+30 -> 10, 18.03 -> 18, 11 -> 11 etc.
+ String version = System.getProperty("java.version");
+ major = Integer.parseInt(version.split("\\.")[0].split("-")[0]);
+} catch (NumberFormatException ignored) {
+}
+IS_JAVA9_OR_ABOVE = major >= 9;
+ }
+
+ public static boolean isJava9OrAbove() {
+return IS_JAVA9_OR_ABOVE;
+ }
--- End diff --
Hadoop trunk is java 8+ only.
* nobody will backport your patch to branch-2; its using Java 8 classes.
* someone could backport an `isJavaVersionAtLeast()` method to branch-2, in
which case, they get to extend it.
> Use java.util.zip.CRC32C for Java 9 and above
> -
>
> Key: HADOOP-15033
> URL: https://issues.apache.org/jira/browse/HADOOP-15033
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, util
>Affects Versions: 3.0.0
>Reporter: Dmitry Chuyko
> Labels: Java9, common, jdk9
> Attachments: HADOOP-15033.001.patch, HADOOP-15033.001.patch,
> HADOOP-15033.002.patch, HADOOP-15033.003.patch, HADOOP-15033.003.patch,
> HADOOP-15033.004.patch
>
>
> java.util.zip.CRC32C implementation is available since Java 9.
> https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html
> Platform specific assembler intrinsics make it more effective than any pure
> Java implementation.
> Hadoop is compiled against Java 8 but class constructor may be accessible
> with method handle on 9 to instances implementing Checksum in runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15131) ADL TestAdlContractRenameLive.testRenameFileUnderFile failing
Steve Loughran created HADOOP-15131:
---
Summary: ADL TestAdlContractRenameLive.testRenameFileUnderFile
failing
Key: HADOOP-15131
URL: https://issues.apache.org/jira/browse/HADOOP-15131
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/adl, test
Affects Versions: 3.1.0
Reporter: Steve Loughran
Priority: Minor
{{TestAdlContractRenameLive.testRenameFileUnderFile}} raises an
AccessControlException when you try to rename something under a file.
Failure is a good outcome, though rename() normally likes to fail silently with
an error code of "false" in such a situation
Options:
# catch the specific exception, look for the text "Forbidden. Parent path is
not a folder." and downgrade to a fail.
# declare this is a valid outcome, override the test case to expect it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15131) ADL TestAdlContractRenameLive.testRenameFileUnderFile failing
[
https://issues.apache.org/jira/browse/HADOOP-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297025#comment-16297025
]
Steve Loughran commented on HADOOP-15131:
-
{code}
[ERROR] Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 16.634
s <<< FAILURE! - in org.apache.hadoop.fs.adl.live.TestAdlContractRenameLive
[ERROR]
testRenameFileUnderFile(org.apache.hadoop.fs.adl.live.TestAdlContractRenameLive)
Time elapsed: 1.348 s <<< ERROR!
org.apache.hadoop.security.AccessControlException: RENAME failed with error
0x83090c88 (Forbidden. Parent path is not a folder.).
[8278d388-9e16-49b0-842c-e123a781d8fd][2017-12-19T07:58:17.8952425-08:00]
[ServerRequestId:8278d388-9e16-49b0-842c-e123a781d8fd]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
com.microsoft.azure.datalake.store.ADLStoreClient.getRemoteException(ADLStoreClient.java:1167)
at
com.microsoft.azure.datalake.store.ADLStoreClient.getExceptionFromResponse(ADLStoreClient.java:1132)
at
com.microsoft.azure.datalake.store.ADLStoreClient.rename(ADLStoreClient.java:673)
at
com.microsoft.azure.datalake.store.ADLStoreClient.rename(ADLStoreClient.java:642)
at org.apache.hadoop.fs.adl.AdlFileSystem.rename(AdlFileSystem.java:515)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.rename(AbstractFSContractTestBase.java:372)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.expectRenameUnderFileFails(AbstractContractRenameTest.java:327)
at
org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameFileUnderFile(AbstractContractRenameTest.java:299)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
> ADL TestAdlContractRenameLive.testRenameFileUnderFile failing
> -
>
> Key: HADOOP-15131
> URL: https://issues.apache.org/jira/browse/HADOOP-15131
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl, test
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> {{TestAdlContractRenameLive.testRenameFileUnderFile}} raises an
> AccessControlException when you try to rename something under a file.
> Failure is a good outcome, though rename() normally likes to fail silently
> with an error code of "false" in such a situation
> Options:
> # catch the specific exception, look for the text "Forbidden. Parent path is
> not a folder." and downgrade to a fail.
> # declare this is a valid outcome, override the test case to expect it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296888#comment-16296888
]
Rushabh S Shah edited comment on HADOOP-15129 at 12/19/17 2:40 PM:
---
Isn't this dupe of HDFS-8068 ?
Also it is related to HADOOP-12125.
was (Author: shahrs87):
Isn't this dupe of HDFS-8068 ?
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna see the Datanode log: "Namenode...remains
> unresolved" -- referring to the fact that DNS lookup failed.
> {code:java}
> 2017-11-02 16:01:55,115 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Refresh request received for nameservices: null
> 2017-11-02 16:01:55,153 WARN org.apache.hadoop.hdfs.DFSUtilClient: Namenode
> for null remains unresolved for ID null. Check your hdfs-site.xml file to
> ensure namenodes are configured properly.
> 2017-11-02 16:01:55,156 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Starting BPOfferServices for nameservices:
> 2017-11-02 16:01:55,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block pool (Datanode Uuid unassigned) service to
> cluster-32f5-m:8020 starting to offer service
> {code}
> The Datanode then proceeds to use this unresolved address, as it may work if
> the DN is configured to use a proxy. Since I'm not using a proxy, it forever
> prints out this message:
> {code:java}
> 2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:45,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:50,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:55,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:14:00,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> {code}
> Unfortunately, the log doesn't contain the exception that triggered it, but
> the culprit is actually in IPC Client:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L444.
> This line was introduced in https://issues.apache.org/jira/browse/HADOOP-487
> to give a clear error message when somebody mispells an address.
> However, the fix in HADOOP-7472 doesn't apply here, because that code happens
> in Client#getConnection after the Connection is constructed.
> My proposed fix (will attach a patch) is to move this exception out of the
> constructor and into a place that will trigger HADOOP-7472's logic to
> re-resolve addresses. If the DNS failure was temporary, this will allow the
> connection to succeed. If not, the connection will fail after ipc client
> retries (default 10 seconds worth of retries).
> I want to fix this in ipc client rather than just in Datanode startup, as
> this fixes temporary DNS issues for all of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296888#comment-16296888
]
Rushabh S Shah commented on HADOOP-15129:
-
Isn't this dupe of HDFS-8068 ?
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna see the Datanode log: "Namenode...remains
> unresolved" -- referring to the fact that DNS lookup failed.
> {code:java}
> 2017-11-02 16:01:55,115 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Refresh request received for nameservices: null
> 2017-11-02 16:01:55,153 WARN org.apache.hadoop.hdfs.DFSUtilClient: Namenode
> for null remains unresolved for ID null. Check your hdfs-site.xml file to
> ensure namenodes are configured properly.
> 2017-11-02 16:01:55,156 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Starting BPOfferServices for nameservices:
> 2017-11-02 16:01:55,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block pool (Datanode Uuid unassigned) service to
> cluster-32f5-m:8020 starting to offer service
> {code}
> The Datanode then proceeds to use this unresolved address, as it may work if
> the DN is configured to use a proxy. Since I'm not using a proxy, it forever
> prints out this message:
> {code:java}
> 2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:45,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:50,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:55,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:14:00,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> {code}
> Unfortunately, the log doesn't contain the exception that triggered it, but
> the culprit is actually in IPC Client:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L444.
> This line was introduced in https://issues.apache.org/jira/browse/HADOOP-487
> to give a clear error message when somebody mispells an address.
> However, the fix in HADOOP-7472 doesn't apply here, because that code happens
> in Client#getConnection after the Connection is constructed.
> My proposed fix (will attach a patch) is to move this exception out of the
> constructor and into a place that will trigger HADOOP-7472's logic to
> re-resolve addresses. If the DNS failure was temporary, this will allow the
> connection to succeed. If not, the connection will fail after ipc client
> retries (default 10 seconds worth of retries).
> I want to fix this in ipc client rather than just in Datanode startup, as
> this fixes temporary DNS issues for all of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13809) hive: 'java.lang.IllegalStateException(zip file closed)'
[
https://issues.apache.org/jira/browse/HADOOP-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296867#comment-16296867
]
Steve Loughran commented on HADOOP-13809:
-
Configuration XML parsing has been completely redone for 2.0+, moving from a
DOM to streaming parser. This is probably going to be a "Cannot Reproduce".
w.r.t CDH, you'll have to talk them I'm afraid
> hive: 'java.lang.IllegalStateException(zip file closed)'
>
>
> Key: HADOOP-13809
> URL: https://issues.apache.org/jira/browse/HADOOP-13809
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 2.8.0
>Reporter: Adriano
>
> Randomly some of the hive queries are failing with the below exception on
> HS2:
> {code}
> 2016-11-07 02:36:40,996 ERROR org.apache.hadoop.hive.ql.exec.Task:
> [HiveServer2-Background-Pool: Thread-1823748]: Ended Job =
> job_1478336955303_31030 with exception 'java.lang.IllegalStateException(zip
> file
> closed)'
> java.lang.IllegalStateException: zip file closed
> at java.util.zip.ZipFile.ensureOpen(ZipFile.java:634)
> at java.util.zip.ZipFile.getEntry(ZipFile.java:305)
> at java.util.jar.JarFile.getEntry(JarFile.java:227)
> at sun.net.www.protocol.jar.URLJarFile.getEntry(URLJarFile.java:128)
> at
> sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:132)
> at
> sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
>
> at
> java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:233)
> at javax.xml.parsers.SecuritySupport$4.run(SecuritySupport.java:94)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.xml.parsers.SecuritySupport.getResourceAsStream(SecuritySupport.java:87)
>
> at
> javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:283)
>
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
>
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:982)
> at
> org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2032)
> at org.apache.hadoop.mapred.JobConf.(JobConf.java:484)
> at org.apache.hadoop.mapred.JobConf.(JobConf.java:474)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:210)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:596)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:594)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
>
> at
> org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:594)
> at
> org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:665)
> at
> org.apache.hadoop.mapred.JobClient.getReduceTaskReports(JobClient.java:689)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:272)
>
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549)
>
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:435)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1770)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1527)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1306)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1115)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:178)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:72)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:232)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at
[jira] [Updated] (HADOOP-13809) hive: 'java.lang.IllegalStateException(zip file closed)'
[
https://issues.apache.org/jira/browse/HADOOP-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13809:
Affects Version/s: (was: 3.0.0-alpha1)
> hive: 'java.lang.IllegalStateException(zip file closed)'
>
>
> Key: HADOOP-13809
> URL: https://issues.apache.org/jira/browse/HADOOP-13809
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 2.8.0
>Reporter: Adriano
>
> Randomly some of the hive queries are failing with the below exception on
> HS2:
> {code}
> 2016-11-07 02:36:40,996 ERROR org.apache.hadoop.hive.ql.exec.Task:
> [HiveServer2-Background-Pool: Thread-1823748]: Ended Job =
> job_1478336955303_31030 with exception 'java.lang.IllegalStateException(zip
> file
> closed)'
> java.lang.IllegalStateException: zip file closed
> at java.util.zip.ZipFile.ensureOpen(ZipFile.java:634)
> at java.util.zip.ZipFile.getEntry(ZipFile.java:305)
> at java.util.jar.JarFile.getEntry(JarFile.java:227)
> at sun.net.www.protocol.jar.URLJarFile.getEntry(URLJarFile.java:128)
> at
> sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:132)
> at
> sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
>
> at
> java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:233)
> at javax.xml.parsers.SecuritySupport$4.run(SecuritySupport.java:94)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.xml.parsers.SecuritySupport.getResourceAsStream(SecuritySupport.java:87)
>
> at
> javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:283)
>
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
>
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:982)
> at
> org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2032)
> at org.apache.hadoop.mapred.JobConf.(JobConf.java:484)
> at org.apache.hadoop.mapred.JobConf.(JobConf.java:474)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:210)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:596)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:594)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
>
> at
> org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:594)
> at
> org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:665)
> at
> org.apache.hadoop.mapred.JobClient.getReduceTaskReports(JobClient.java:689)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:272)
>
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549)
>
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:435)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1770)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1527)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1306)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1115)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:178)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:72)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:232)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
>
> at
>
[jira] [Commented] (HADOOP-15129) Datanode caches namenode DNS lookup failure and cannot startup
[
https://issues.apache.org/jira/browse/HADOOP-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296865#comment-16296865
]
Steve Loughran commented on HADOOP-15129:
-
the IPC is very sensitive code, as DN/NN comms, so this is going to need review
by people who care about that, maybe [~arpitagarwal].
to me, production code looks fine, but we need reviewers from HDFS too.
Test wise,
* use LambdaTestUtils.intercept() & java-8 lambda expressions for those tests
which throw exceptions; if your lambda returns any object, toString() will be
caused on it & the text used in the exception...this can be useful
* and as Client is autocloseable, you can use try-with-resources to manage its
lifecycle
Test failures unrelated & covered elsewhere
> Datanode caches namenode DNS lookup failure and cannot startup
> --
>
> Key: HADOOP-15129
> URL: https://issues.apache.org/jira/browse/HADOOP-15129
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.8.2
> Environment: Google Compute Engine.
> I'm using Java 8, Debian 8, Hadoop 2.8.2.
>Reporter: Karthik Palaniappan
>Priority: Minor
> Attachments: HADOOP-15129.001.patch
>
>
> On startup, the Datanode creates an InetSocketAddress to register with each
> namenode. Though there are retries on connection failure throughout the
> stack, the same InetSocketAddress is reused.
> InetSocketAddress is an interesting class, because it resolves DNS names to
> IP addresses on construction, and it is never refreshed. Hadoop re-creates an
> InetSocketAddress in some cases just in case the remote IP has changed for a
> particular DNS name: https://issues.apache.org/jira/browse/HADOOP-7472.
> Anyway, on startup, you cna see the Datanode log: "Namenode...remains
> unresolved" -- referring to the fact that DNS lookup failed.
> {code:java}
> 2017-11-02 16:01:55,115 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Refresh request received for nameservices: null
> 2017-11-02 16:01:55,153 WARN org.apache.hadoop.hdfs.DFSUtilClient: Namenode
> for null remains unresolved for ID null. Check your hdfs-site.xml file to
> ensure namenodes are configured properly.
> 2017-11-02 16:01:55,156 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Starting BPOfferServices for nameservices:
> 2017-11-02 16:01:55,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Block pool (Datanode Uuid unassigned) service to
> cluster-32f5-m:8020 starting to offer service
> {code}
> The Datanode then proceeds to use this unresolved address, as it may work if
> the DN is configured to use a proxy. Since I'm not using a proxy, it forever
> prints out this message:
> {code:java}
> 2017-12-15 00:13:40,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:45,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:50,712 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:13:55,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> 2017-12-15 00:14:00,713 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Problem connecting to server: cluster-32f5-m:8020
> {code}
> Unfortunately, the log doesn't contain the exception that triggered it, but
> the culprit is actually in IPC Client:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L444.
> This line was introduced in https://issues.apache.org/jira/browse/HADOOP-487
> to give a clear error message when somebody mispells an address.
> However, the fix in HADOOP-7472 doesn't apply here, because that code happens
> in Client#getConnection after the Connection is constructed.
> My proposed fix (will attach a patch) is to move this exception out of the
> constructor and into a place that will trigger HADOOP-7472's logic to
> re-resolve addresses. If the DNS failure was temporary, this will allow the
> connection to succeed. If not, the connection will fail after ipc client
> retries (default 10 seconds worth of retries).
> I want to fix this in ipc client rather than just in Datanode startup, as
> this fixes temporary DNS issues for all of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[
https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296844#comment-16296844
]
ASF GitHub Bot commented on HADOOP-15033:
-
Github user dchuyko commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/291#discussion_r157764279
--- Diff:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
---
@@ -89,6 +89,22 @@ public static boolean isJava7OrAbove() {
return true;
}
+ private static final boolean IS_JAVA9_OR_ABOVE;
+ static {
+int major = 0;
+try {
+ // 1.8.0 -> 1, 10-ea+30 -> 10, 18.03 -> 18, 11 -> 11 etc.
+ String version = System.getProperty("java.version");
+ major = Integer.parseInt(version.split("\\.")[0].split("-")[0]);
+} catch (NumberFormatException ignored) {
+}
+IS_JAVA9_OR_ABOVE = major >= 9;
+ }
+
+ public static boolean isJava9OrAbove() {
+return IS_JAVA9_OR_ABOVE;
+ }
--- End diff --
Yes, I had exactly the same in mind. Just haven't realized we may now
assume 8 as lowest version. Will it still be true if someone decides to
backport the fix to previous Hadoop versions?
> Use java.util.zip.CRC32C for Java 9 and above
> -
>
> Key: HADOOP-15033
> URL: https://issues.apache.org/jira/browse/HADOOP-15033
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, util
>Affects Versions: 3.0.0
>Reporter: Dmitry Chuyko
> Labels: Java9, common, jdk9
> Attachments: HADOOP-15033.001.patch, HADOOP-15033.001.patch,
> HADOOP-15033.002.patch, HADOOP-15033.003.patch, HADOOP-15033.003.patch,
> HADOOP-15033.004.patch
>
>
> java.util.zip.CRC32C implementation is available since Java 9.
> https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html
> Platform specific assembler intrinsics make it more effective than any pure
> Java implementation.
> Hadoop is compiled against Java 8 but class constructor may be accessible
> with method handle on 9 to instances implementing Checksum in runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[
https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296841#comment-16296841
]
ASF GitHub Bot commented on HADOOP-15033:
-
Github user dchuyko commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/291#discussion_r157763644
--- Diff:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java
---
@@ -70,6 +74,24 @@ public static Type valueOf(int id) {
}
}
+ private static final MethodHandle NEW_CRC32C_MH;
+ static {
--- End diff --
Agree, logging would go better. I was trying to avoid silent failure here.
> Use java.util.zip.CRC32C for Java 9 and above
> -
>
> Key: HADOOP-15033
> URL: https://issues.apache.org/jira/browse/HADOOP-15033
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, util
>Affects Versions: 3.0.0
>Reporter: Dmitry Chuyko
> Labels: Java9, common, jdk9
> Attachments: HADOOP-15033.001.patch, HADOOP-15033.001.patch,
> HADOOP-15033.002.patch, HADOOP-15033.003.patch, HADOOP-15033.003.patch,
> HADOOP-15033.004.patch
>
>
> java.util.zip.CRC32C implementation is available since Java 9.
> https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html
> Platform specific assembler intrinsics make it more effective than any pure
> Java implementation.
> Hadoop is compiled against Java 8 but class constructor may be accessible
> with method handle on 9 to instances implementing Checksum in runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[
https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296837#comment-16296837
]
ASF GitHub Bot commented on HADOOP-15033:
-
Github user dchuyko commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/291#discussion_r157763230
--- Diff:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java
---
@@ -78,6 +100,16 @@ public static Checksum newCrc32() {
return new CRC32();
}
+ public static Checksum newCrc32C() {
+try {
+ return Shell.isJava9OrAbove() ? (Checksum) NEW_CRC32C_MH.invoke()
+ : new PureJavaCrc32C();
+} catch (Throwable e) {
+ // Should not reach here.
+ throw new RuntimeException(e);
--- End diff --
Got it, this will be more clean.
> Use java.util.zip.CRC32C for Java 9 and above
> -
>
> Key: HADOOP-15033
> URL: https://issues.apache.org/jira/browse/HADOOP-15033
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, util
>Affects Versions: 3.0.0
>Reporter: Dmitry Chuyko
> Labels: Java9, common, jdk9
> Attachments: HADOOP-15033.001.patch, HADOOP-15033.001.patch,
> HADOOP-15033.002.patch, HADOOP-15033.003.patch, HADOOP-15033.003.patch,
> HADOOP-15033.004.patch
>
>
> java.util.zip.CRC32C implementation is available since Java 9.
> https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html
> Platform specific assembler intrinsics make it more effective than any pure
> Java implementation.
> Hadoop is compiled against Java 8 but class constructor may be accessible
> with method handle on 9 to instances implementing Checksum in runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[
https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296805#comment-16296805
]
ASF GitHub Bot commented on HADOOP-15033:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/291#discussion_r157757732
--- Diff:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
---
@@ -89,6 +89,22 @@ public static boolean isJava7OrAbove() {
return true;
}
+ private static final boolean IS_JAVA9_OR_ABOVE;
+ static {
+int major = 0;
+try {
+ // 1.8.0 -> 1, 10-ea+30 -> 10, 18.03 -> 18, 11 -> 11 etc.
+ String version = System.getProperty("java.version");
+ major = Integer.parseInt(version.split("\\.")[0].split("-")[0]);
+} catch (NumberFormatException ignored) {
+}
+IS_JAVA9_OR_ABOVE = major >= 9;
+ }
+
+ public static boolean isJava9OrAbove() {
+return IS_JAVA9_OR_ABOVE;
+ }
--- End diff --
Given we've already got isJava7OrAbove() tagged as deprecated, we should
stop repeating the same mistake.
How about some: `boolean IsJavaVersionAtLeast(int)` which can do checks for
versions, store this parsed major in a private static (set the default value to
8, obviously) field, then have the new predicate return true if the param is >=
the cached value
> Use java.util.zip.CRC32C for Java 9 and above
> -
>
> Key: HADOOP-15033
> URL: https://issues.apache.org/jira/browse/HADOOP-15033
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, util
>Affects Versions: 3.0.0
>Reporter: Dmitry Chuyko
> Labels: Java9, common, jdk9
> Attachments: HADOOP-15033.001.patch, HADOOP-15033.001.patch,
> HADOOP-15033.002.patch, HADOOP-15033.003.patch, HADOOP-15033.003.patch,
> HADOOP-15033.004.patch
>
>
> java.util.zip.CRC32C implementation is available since Java 9.
> https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html
> Platform specific assembler intrinsics make it more effective than any pure
> Java implementation.
> Hadoop is compiled against Java 8 but class constructor may be accessible
> with method handle on 9 to instances implementing Checksum in runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[
https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296800#comment-16296800
]
ASF GitHub Bot commented on HADOOP-15033:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/291#discussion_r157756596
--- Diff:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java
---
@@ -70,6 +74,24 @@ public static Type valueOf(int id) {
}
}
+ private static final MethodHandle NEW_CRC32C_MH;
+ static {
--- End diff --
I'd like this to all be on demand, so that the introspection is only done
if needed, which allows the exception to be thrown at the right point.
Also, could we just back off from failing here at all, just log, track that
the operation failed (so no attempt is repeated), and then fall back to the
older mechanisms. I don't want things to overreact here
> Use java.util.zip.CRC32C for Java 9 and above
> -
>
> Key: HADOOP-15033
> URL: https://issues.apache.org/jira/browse/HADOOP-15033
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, util
>Affects Versions: 3.0.0
>Reporter: Dmitry Chuyko
> Labels: Java9, common, jdk9
> Attachments: HADOOP-15033.001.patch, HADOOP-15033.001.patch,
> HADOOP-15033.002.patch, HADOOP-15033.003.patch, HADOOP-15033.003.patch,
> HADOOP-15033.004.patch
>
>
> java.util.zip.CRC32C implementation is available since Java 9.
> https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html
> Platform specific assembler intrinsics make it more effective than any pure
> Java implementation.
> Hadoop is compiled against Java 8 but class constructor may be accessible
> with method handle on 9 to instances implementing Checksum in runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15033) Use java.util.zip.CRC32C for Java 9 and above
[
https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296796#comment-16296796
]
ASF GitHub Bot commented on HADOOP-15033:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/291#discussion_r157755880
--- Diff:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DataChecksum.java
---
@@ -78,6 +100,16 @@ public static Checksum newCrc32() {
return new CRC32();
}
+ public static Checksum newCrc32C() {
+try {
+ return Shell.isJava9OrAbove() ? (Checksum) NEW_CRC32C_MH.invoke()
+ : new PureJavaCrc32C();
+} catch (Throwable e) {
+ // Should not reach here.
+ throw new RuntimeException(e);
--- End diff --
RuntimeException shouldn't be wrapped; leave the other Throwables alone.
So: `catch(RuntimeException e) { throw e;}`, then catch and wrap Exception only,
> Use java.util.zip.CRC32C for Java 9 and above
> -
>
> Key: HADOOP-15033
> URL: https://issues.apache.org/jira/browse/HADOOP-15033
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, util
>Affects Versions: 3.0.0
>Reporter: Dmitry Chuyko
> Labels: Java9, common, jdk9
> Attachments: HADOOP-15033.001.patch, HADOOP-15033.001.patch,
> HADOOP-15033.002.patch, HADOOP-15033.003.patch, HADOOP-15033.003.patch,
> HADOOP-15033.004.patch
>
>
> java.util.zip.CRC32C implementation is available since Java 9.
> https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html
> Platform specific assembler intrinsics make it more effective than any pure
> Java implementation.
> Hadoop is compiled against Java 8 but class constructor may be accessible
> with method handle on 9 to instances implementing Checksum in runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15128) TestViewFileSystem tests are broken in trunk
[
https://issues.apache.org/jira/browse/HADOOP-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296772#comment-16296772
]
Steve Loughran commented on HADOOP-15128:
-
We've had problems with RawLocalFileSystem#loadPermissionInfo() being a
performance killer when it execs local fs operations, especially on Windows:
HADOOP-14600.
What is toString being used for?
* If is being logged as the output of operations like {{hadoop fs -ls}} then
we're duty bound by way of the compat guidelines to not break the format of the
output. I believe the patch is safe there, other than the API changes
* If its just being logged in our info/debug strings, calling loadParmissions
is a very expensive operation to undertake in some filesystems, and, as noted,
it can fail. And, because it changes the state of the object, it means that log
statements will have side effects when enabled. I don't want to be in the
situation where logging @ debug makes code work, and turning it off fixes it;
that's the bane of C/C++ debug macros.
Is there a way to do a fix which doesn't force an eval in a normal toString,
but only when needed for logging to the console? I'm sure we've done it with
other classes where we ended up adding a "for console output" toString
equivalent. Though I like fileStatus to stay the same, given its ubiquity
> TestViewFileSystem tests are broken in trunk
>
>
> Key: HADOOP-15128
> URL: https://issues.apache.org/jira/browse/HADOOP-15128
> Project: Hadoop Common
> Issue Type: Bug
> Components: viewfs
>Affects Versions: 3.1.0
>Reporter: Anu Engineer
>Assignee: Hanisha Koneru
>
> The fix in Hadoop-10054 seems to have caused a test failure. Please take a
> look. Thanks [~eyang] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13809) hive: 'java.lang.IllegalStateException(zip file closed)'
[
https://issues.apache.org/jira/browse/HADOOP-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296701#comment-16296701
]
jiangxiyang commented on HADOOP-13809:
--
Any progress on this bug?
We met the same problem, with OpenJDK 1.7.0_79 and CDH 5.13
> hive: 'java.lang.IllegalStateException(zip file closed)'
>
>
> Key: HADOOP-13809
> URL: https://issues.apache.org/jira/browse/HADOOP-13809
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Adriano
>
> Randomly some of the hive queries are failing with the below exception on
> HS2:
> {code}
> 2016-11-07 02:36:40,996 ERROR org.apache.hadoop.hive.ql.exec.Task:
> [HiveServer2-Background-Pool: Thread-1823748]: Ended Job =
> job_1478336955303_31030 with exception 'java.lang.IllegalStateException(zip
> file
> closed)'
> java.lang.IllegalStateException: zip file closed
> at java.util.zip.ZipFile.ensureOpen(ZipFile.java:634)
> at java.util.zip.ZipFile.getEntry(ZipFile.java:305)
> at java.util.jar.JarFile.getEntry(JarFile.java:227)
> at sun.net.www.protocol.jar.URLJarFile.getEntry(URLJarFile.java:128)
> at
> sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:132)
> at
> sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
>
> at
> java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:233)
> at javax.xml.parsers.SecuritySupport$4.run(SecuritySupport.java:94)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> javax.xml.parsers.SecuritySupport.getResourceAsStream(SecuritySupport.java:87)
>
> at
> javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:283)
>
> at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
> at
> javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
>
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2526)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:982)
> at
> org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2032)
> at org.apache.hadoop.mapred.JobConf.(JobConf.java:484)
> at org.apache.hadoop.mapred.JobConf.(JobConf.java:474)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:210)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:596)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:594)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
>
> at
> org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:594)
> at
> org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:665)
> at
> org.apache.hadoop.mapred.JobClient.getReduceTaskReports(JobClient.java:689)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:272)
>
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:549)
>
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:435)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1770)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1527)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1306)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1115)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:178)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:72)
>
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:232)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
>
