[jira] [Updated] (HADOOP-9836) Token definition and API
[ https://issues.apache.org/jira/browse/HADOOP-9836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianyou Li updated HADOOP-9836: --- Issue Type: Sub-task (was: Task) Parent: HADOOP-9392 Token definition and API Key: HADOOP-9836 URL: https://issues.apache.org/jira/browse/HADOOP-9836 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 3.0.0 Reporter: Yi Liu Labels: Rhino We need to define common token attributes and APIs for TokenAuth framework which makes the arbitrary token format can be adopted into the framework. This JIRA is a sub-task of TokenAuth framework. Common token properties, APIs and facilities that identity/access token requires will be defined. In this JIRA, we'll: • Define Token generation API, includes Token serialization/deserialization, Token encryption/sign and Token revoke/expire/renew. • Define Token validation API, includes Token decryption/verify and Token check(timestamp, audience, etc) • Define Token Attribute API, includes attributes setting, query and so on. • Define required attributes and optional attributes for identity token and access token. • Implement Token Utilities, such as print/debug. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10030) FsShell -put/copyFromLocal should support Windows local path
[
https://issues.apache.org/jira/browse/HADOOP-10030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790246#comment-13790246
]
Hudson commented on HADOOP-10030:
-
SUCCESS: Integrated in Hadoop-Yarn-trunk #357 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/357/])
HADOOP-10030. FsShell -put/copyFromLocal should support Windows local path.
Contributed by Chuan Liu. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1530462)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
FsShell -put/copyFromLocal should support Windows local path
Key: HADOOP-10030
URL: https://issues.apache.org/jira/browse/HADOOP-10030
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Fix For: 3.0.0, 2.2.1
Attachments: HADOOP-10030.patch
In current trunk code, running 'hadoop fs -put c:\windows\path /hdfs/path'
will lead to an error {{put: unexpected URISyntaxException}}. This is a
regression to the 1.0 fs shell commands. FsShell put and copyFromLocal should
support Windows local path formats as localSrc argument to the two commands.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9199) Cover package org.apache.hadoop.io with unit tests
[ https://issues.apache.org/jira/browse/HADOOP-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790247#comment-13790247 ] Hudson commented on HADOOP-9199: SUCCESS: Integrated in Hadoop-Yarn-trunk #357 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/357/]) HADOOP-9199. Cover package org.apache.hadoop.io with unit tests (Andrey Klochkov via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1530404) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestArrayFile.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestArrayWritable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestBloomMapFile.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestBooleanWritable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestBytesWritable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestEnumSetWritable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestMapFile.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSetFile.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java Cover package org.apache.hadoop.io with unit tests -- Key: HADOOP-9199 URL: https://issues.apache.org/jira/browse/HADOOP-9199 Project: Hadoop Common Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Vadim Bondarev Assignee: Andrey Klochkov Fix For: 3.0.0, 2.3.0 Attachments: HADOOP-9199-branch-0.23-a.patch, HADOOP-9199-branch-0.23-b.patch, HADOOP-9199-branch-0.23-c.patch, HADOOP-9199-branch-0.23-e.patch, HADOOP-9199-branch-2-a.patch, HADOOP-9199-branch-2-b.patch, HADOOP-9199-branch-2-c.patch, HADOOP-9199-branch-2-e.patch, HADOOP-9199-trunk-a.patch, HADOOP-9199-trunk-b.patch, HADOOP-9199-trunk-c.patch, HADOOP-9199-trunk-e.patch, HADOOP-9199-trunk--n6.patch, HADOOP-9199-trunk--n7.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10030) FsShell -put/copyFromLocal should support Windows local path
[
https://issues.apache.org/jira/browse/HADOOP-10030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790323#comment-13790323
]
Hudson commented on HADOOP-10030:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1573 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1573/])
HADOOP-10030. FsShell -put/copyFromLocal should support Windows local path.
Contributed by Chuan Liu. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1530462)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
FsShell -put/copyFromLocal should support Windows local path
Key: HADOOP-10030
URL: https://issues.apache.org/jira/browse/HADOOP-10030
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Fix For: 3.0.0, 2.2.1
Attachments: HADOOP-10030.patch
In current trunk code, running 'hadoop fs -put c:\windows\path /hdfs/path'
will lead to an error {{put: unexpected URISyntaxException}}. This is a
regression to the 1.0 fs shell commands. FsShell put and copyFromLocal should
support Windows local path formats as localSrc argument to the two commands.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9199) Cover package org.apache.hadoop.io with unit tests
[ https://issues.apache.org/jira/browse/HADOOP-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790324#comment-13790324 ] Hudson commented on HADOOP-9199: FAILURE: Integrated in Hadoop-Mapreduce-trunk #1573 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1573/]) HADOOP-9199. Cover package org.apache.hadoop.io with unit tests (Andrey Klochkov via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1530404) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestArrayFile.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestArrayWritable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestBloomMapFile.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestBooleanWritable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestBytesWritable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestEnumSetWritable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestMapFile.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSetFile.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java Cover package org.apache.hadoop.io with unit tests -- Key: HADOOP-9199 URL: https://issues.apache.org/jira/browse/HADOOP-9199 Project: Hadoop Common Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Vadim Bondarev Assignee: Andrey Klochkov Fix For: 3.0.0, 2.3.0 Attachments: HADOOP-9199-branch-0.23-a.patch, HADOOP-9199-branch-0.23-b.patch, HADOOP-9199-branch-0.23-c.patch, HADOOP-9199-branch-0.23-e.patch, HADOOP-9199-branch-2-a.patch, HADOOP-9199-branch-2-b.patch, HADOOP-9199-branch-2-c.patch, HADOOP-9199-branch-2-e.patch, HADOOP-9199-trunk-a.patch, HADOOP-9199-trunk-b.patch, HADOOP-9199-trunk-c.patch, HADOOP-9199-trunk-e.patch, HADOOP-9199-trunk--n6.patch, HADOOP-9199-trunk--n7.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10030) FsShell -put/copyFromLocal should support Windows local path
[
https://issues.apache.org/jira/browse/HADOOP-10030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790350#comment-13790350
]
Hudson commented on HADOOP-10030:
-
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1547 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1547/])
HADOOP-10030. FsShell -put/copyFromLocal should support Windows local path.
Contributed by Chuan Liu. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1530462)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java
FsShell -put/copyFromLocal should support Windows local path
Key: HADOOP-10030
URL: https://issues.apache.org/jira/browse/HADOOP-10030
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Fix For: 3.0.0, 2.2.1
Attachments: HADOOP-10030.patch
In current trunk code, running 'hadoop fs -put c:\windows\path /hdfs/path'
will lead to an error {{put: unexpected URISyntaxException}}. This is a
regression to the 1.0 fs shell commands. FsShell put and copyFromLocal should
support Windows local path formats as localSrc argument to the two commands.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9199) Cover package org.apache.hadoop.io with unit tests
[ https://issues.apache.org/jira/browse/HADOOP-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790351#comment-13790351 ] Hudson commented on HADOOP-9199: SUCCESS: Integrated in Hadoop-Hdfs-trunk #1547 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1547/]) HADOOP-9199. Cover package org.apache.hadoop.io with unit tests (Andrey Klochkov via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1530404) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestArrayFile.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestArrayWritable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestBloomMapFile.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestBooleanWritable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestBytesWritable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestEnumSetWritable.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestMapFile.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSetFile.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java Cover package org.apache.hadoop.io with unit tests -- Key: HADOOP-9199 URL: https://issues.apache.org/jira/browse/HADOOP-9199 Project: Hadoop Common Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Vadim Bondarev Assignee: Andrey Klochkov Fix For: 3.0.0, 2.3.0 Attachments: HADOOP-9199-branch-0.23-a.patch, HADOOP-9199-branch-0.23-b.patch, HADOOP-9199-branch-0.23-c.patch, HADOOP-9199-branch-0.23-e.patch, HADOOP-9199-branch-2-a.patch, HADOOP-9199-branch-2-b.patch, HADOOP-9199-branch-2-c.patch, HADOOP-9199-branch-2-e.patch, HADOOP-9199-trunk-a.patch, HADOOP-9199-trunk-b.patch, HADOOP-9199-trunk-c.patch, HADOOP-9199-trunk-e.patch, HADOOP-9199-trunk--n6.patch, HADOOP-9199-trunk--n7.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10036) RetryInvocationHandler should recognise that there is no point retrying to auth failures
[
https://issues.apache.org/jira/browse/HADOOP-10036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790380#comment-13790380
]
Steve Loughran commented on HADOOP-10036:
-
log
{code}
2013-10-09 15:17:46,631 [main] WARN ipc.Client - Exception encountered while
connecting to the server : java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
2013-10-09 15:17:46,632 [main] ERROR security.UserGroupInformation -
PriviledgedActionException as:stevel@HOME (auth:KERBEROS)
cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
2013-10-09 15:18:16,634 [main] WARN ipc.Client - Exception encountered while
connecting to the server : java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
2013-10-09 15:18:16,634 [main] ERROR security.UserGroupInformation -
PriviledgedActionException as:stevel@HOME (auth:KERBEROS)
cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
2013-10-09 15:18:46,638 [main] WARN ipc.Client - Exception encountered while
connecting to the server : java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
2013-10-09 15:18:46,638 [main] ERROR security.UserGroupInformation -
PriviledgedActionException as:stevel@HOME (auth:KERBEROS)
cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
2013-10-09 15:19:16,641 [main] WARN ipc.Client - Exception encountered while
connecting to the server : java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
2013-10-09 15:19:16,641 [main] ERROR security.UserGroupInformation -
PriviledgedActionException as:stevel@HOME (auth:KERBEROS)
cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
2013-10-09 15:19:46,643 [main] WARN ipc.Client - Exception encountered while
connecting to the server : java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
2013-10-09 15:19:46,643 [main] ERROR security.UserGroupInformation -
PriviledgedActionException as:stevel@HOME (auth:KERBEROS)
cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
2013-10-09 15:20:16,646 [main] WARN ipc.Client - Exception encountered while
connecting to the server : java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
2013-10-09 15:20:16,647 [main] ERROR security.UserGroupInformation -
PriviledgedActionException as:stevel@HOME (auth:KERBEROS)
cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
2013-10-09 15:20:46,649 [main] WARN ipc.Client - Exception encountered while
connecting to the server : java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
2013-10-09 15:20:46,650 [main] ERROR security.UserGroupInformation -
PriviledgedActionException as:stevel@HOME (auth:KERBEROS)
cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to
specify server's Kerberos principal name
{code}
RetryInvocationHandler should recognise that there is no point retrying to
auth failures
Key: HADOOP-10036
URL: https://issues.apache.org/jira/browse/HADOOP-10036
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Affects Versions: 2.2.1
Reporter: Steve Loughran
The {{RetryInvocationHandler}} tries to retry connections, so as to handle
transient failures.
However, auth failures aren't treated as special, so it spins even though the
operation will not succeed with the current configuration
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HADOOP-10036) RetryInvocationHandler should recognise that there is no point retrying to auth failures
Steve Loughran created HADOOP-10036:
---
Summary: RetryInvocationHandler should recognise that there is no
point retrying to auth failures
Key: HADOOP-10036
URL: https://issues.apache.org/jira/browse/HADOOP-10036
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Affects Versions: 2.2.1
Reporter: Steve Loughran
The {{RetryInvocationHandler}} tries to retry connections, so as to handle
transient failures.
However, auth failures aren't treated as special, so it spins even though the
operation will not succeed with the current configuration
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-10036) RetryInvocationHandler should recognise that there is no point retrying to auth failures
[
https://issues.apache.org/jira/browse/HADOOP-10036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790381#comment-13790381
]
Steve Loughran commented on HADOOP-10036:
-
stack
{code}
main prio=10 tid=0xb6809c00 nid=0x45cb waiting on condition [0xb69b6000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.util.ThreadUtil.sleepAtLeastIgnoreInterrupts(ThreadUtil.java:43)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:150)
at com.sun.proxy.$Proxy10.getApplications(Unknown Source)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:237)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:221)
at
org.apache.hadoop.hoya.yarn.client.HoyaClient.listHoyaInstances(HoyaClient.java:1182)
at
org.apache.hadoop.hoya.yarn.client.HoyaClient.actionList(HoyaClient.java:1201)
at
org.apache.hadoop.hoya.yarn.client.HoyaClient.runService(HoyaClient.java:204)
at
org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchService(ServiceLauncher.java:175)
at
org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchServiceRobustly(ServiceLauncher.java:383)
at
org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchServiceAndExit(ServiceLauncher.java:312)
at
org.apache.hadoop.yarn.service.launcher.ServiceLauncher.serviceMain(ServiceLauncher.java:524)
at org.apache.hadoop.hoya.Hoya.main(Hoya.java:49)
{code}
RetryInvocationHandler should recognise that there is no point retrying to
auth failures
Key: HADOOP-10036
URL: https://issues.apache.org/jira/browse/HADOOP-10036
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Affects Versions: 2.2.1
Reporter: Steve Loughran
The {{RetryInvocationHandler}} tries to retry connections, so as to handle
transient failures.
However, auth failures aren't treated as special, so it spins even though the
operation will not succeed with the current configuration
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HADOOP-10036) RetryInvocationHandler should recognise that there is no point retrying to auth failures
[
https://issues.apache.org/jira/browse/HADOOP-10036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-10036:
Priority: Minor (was: Major)
RetryInvocationHandler should recognise that there is no point retrying to
auth failures
Key: HADOOP-10036
URL: https://issues.apache.org/jira/browse/HADOOP-10036
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Affects Versions: 2.2.1
Reporter: Steve Loughran
Priority: Minor
The {{RetryInvocationHandler}} tries to retry connections, so as to handle
transient failures.
However, auth failures aren't treated as special, so it spins even though the
operation will not succeed with the current configuration
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-10036) RetryInvocationHandler should recognise that there is no point retrying to auth failures
[
https://issues.apache.org/jira/browse/HADOOP-10036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790399#comment-13790399
]
Steve Loughran commented on HADOOP-10036:
-
Maybe the action would be recognise that {{IllegalArgumentException}},
{{NullPointerException}} and a few others are not retryable, though that is a
slippery slope
RetryInvocationHandler should recognise that there is no point retrying to
auth failures
Key: HADOOP-10036
URL: https://issues.apache.org/jira/browse/HADOOP-10036
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Affects Versions: 2.2.1
Reporter: Steve Loughran
Priority: Minor
The {{RetryInvocationHandler}} tries to retry connections, so as to handle
transient failures.
However, auth failures aren't treated as special, so it spins even though the
operation will not succeed with the current configuration
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HADOOP-9470) eliminate duplicate FQN tests in different Hadoop modules
[ https://issues.apache.org/jira/browse/HADOOP-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-9470: Resolution: Fixed Fix Version/s: 2.3.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1. Thanks! Committed to trunk and branch-2. eliminate duplicate FQN tests in different Hadoop modules - Key: HADOOP-9470 URL: https://issues.apache.org/jira/browse/HADOOP-9470 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0, 2.3.0 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Fix For: 3.0.0, 2.3.0 Attachments: find-duplicate-fqns.sh, HADOOP-9470-branch-0.23.patch, HADOOP-9470-branch-2--N1.patch, HADOOP-9470-trunk--N1.patch, HADOOP-9470-trunk.patch In different modules of Hadoop project there are tests with identical FQNs (fully qualified name). For example, test with FQN org.apache.hadoop.util.TestRunJar is contained in 2 modules: ./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestRunJar.java ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/util/TestRunJar.java Such situation causes certain problems with test result reporting and other code analysis tools (such as Clover, e.g.) because almost all the tools identify the tests by their Java FQN. So, I suggest to rename all such test classes to avoid duplicate FQNs in different modules. I'm attaching simple shell script that can find all such problematic test classes. Currently Hadoop trunk has 9 such test classes, they are: $ ~/bin/find-duplicate-fqns.sh # Module [./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-classes] has 7 duplicate FQN tests: org.apache.hadoop.ipc.TestSocketFactory org.apache.hadoop.mapred.TestFileOutputCommitter org.apache.hadoop.mapred.TestJobClient org.apache.hadoop.mapred.TestJobConf org.apache.hadoop.mapreduce.lib.output.TestFileOutputCommitter org.apache.hadoop.util.TestReflectionUtils org.apache.hadoop.util.TestRunJar # Module [./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/target/test-classes] has 2 duplicate FQN tests: org.apache.hadoop.yarn.TestRecordFactory org.apache.hadoop.yarn.TestRPCFactories -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9470) eliminate duplicate FQN tests in different Hadoop modules
[ https://issues.apache.org/jira/browse/HADOOP-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790454#comment-13790454 ] Hudson commented on HADOOP-9470: SUCCESS: Integrated in Hadoop-trunk-Commit #4572 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4572/]) HADOOP-9470. eliminate duplicate FQN tests in different Hadoop modules (Ivan A. Veselovsky via daryn) (daryn: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1530667) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/shell/TestHdfsTextCommand.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/shell/TestTextCommand.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/ipc/TestMRCJCSocketFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/ipc/TestSocketFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobConf.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRCJCFileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRCJCFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRCJCJobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRCJCJobConf.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRCJCFileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestMRCJCFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/util/TestMRCJCReflectionUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/util/TestMRCJCRunJar.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/util/TestReflectionUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/util/TestRunJar.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestRPCFactories.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestRecordFactory.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYSCRPCFactories.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYSCRecordFactory.java eliminate duplicate FQN tests in different Hadoop modules - Key: HADOOP-9470 URL: https://issues.apache.org/jira/browse/HADOOP-9470 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0, 2.3.0 Reporter: Ivan A.
[jira] [Commented] (HADOOP-10036) RetryInvocationHandler should recognise that there is no point retrying to auth failures
[
https://issues.apache.org/jira/browse/HADOOP-10036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790524#comment-13790524
]
Steve Loughran commented on HADOOP-10036:
-
... better: have {{SaslRpcClient.getServerPrincipal()}} raise a specific
exception for for an invalid (and irreconcilable) auth configuration problem
which retrying will have no effect whatsoever -there are four possible causes
of this.
RetryInvocationHandler should recognise that there is no point retrying to
auth failures
Key: HADOOP-10036
URL: https://issues.apache.org/jira/browse/HADOOP-10036
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Affects Versions: 2.2.1
Reporter: Steve Loughran
Priority: Minor
The {{RetryInvocationHandler}} tries to retry connections, so as to handle
transient failures.
However, auth failures aren't treated as special, so it spins even though the
operation will not succeed with the current configuration
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HADOOP-10037) s3n read truncated, but doesn't throw exception
David Rosenstrauch created HADOOP-10037: --- Summary: s3n read truncated, but doesn't throw exception Key: HADOOP-10037 URL: https://issues.apache.org/jira/browse/HADOOP-10037 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 2.0.0-alpha Environment: Ubuntu Linux 13.04 running on Amazon EC2 (cc2.8xlarge) Reporter: David Rosenstrauch For months now we've been finding that we've been experiencing frequent data truncation issues when reading from S3 using the s3n:// protocol. I finally was able to gather some debugging output on the issue in a job I ran last night, and so can finally file a bug report. The job I ran last night was on a 16-node cluster (all of them AWS EC2 cc2.8xlarge machines, running Ubuntu 13.04 and Cloudera CDH4.3.0). The job was a Hadoop streaming job, which reads through a large number (i.e., ~55,000) of files on S3, each of them approximately 300K bytes in size. All of the files contain 46 columns of data in each record. But I added in an extra check in my mapper code to count and verify the number of columns in every record - throwing an error and crashing the map task if the column count is wrong. If you look in the attached task logs, you'll see 2 attempts on the same task. The first one fails due to data truncated (i.e., my job intentionally fails the map task due to the current record failing the column count check). The task then gets retried on a different machine and runs to a succesful completion. You can see further evidence of the truncation further down in the task logs, where it displays the count of the records read: the failed task says 32953 records read, while the successful task says 63133. Any idea what the problem might be here and/or how to work around it? This issue is a very common occurrence on our clusters. E.g., in the job I ran last night before I had gone to bed I had already encountered 8 such failuers, and the job was only 10% complete. (~25,000 out of ~250,000 tasks.) I realize that it's common for I/O errors to occur - possibly even frequently - in a large Hadoop job. But I would think that if an I/O failure (like a truncated read) did occur, that something in the underlying infrastructure code (i.e., either in NativeS3FileSystem or in jets3t) should detect the error and throw an IOException accordingly. It shouldn't be up to the calling code to detect such failures, IMO. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-10037) s3n read truncated, but doesn't throw exception
[ https://issues.apache.org/jira/browse/HADOOP-10037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Rosenstrauch updated HADOOP-10037: Attachment: S3ReadSucceeded.html S3ReadFailedOnTruncation.html s3n read truncated, but doesn't throw exception Key: HADOOP-10037 URL: https://issues.apache.org/jira/browse/HADOOP-10037 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 2.0.0-alpha Environment: Ubuntu Linux 13.04 running on Amazon EC2 (cc2.8xlarge) Reporter: David Rosenstrauch Attachments: S3ReadFailedOnTruncation.html, S3ReadSucceeded.html For months now we've been finding that we've been experiencing frequent data truncation issues when reading from S3 using the s3n:// protocol. I finally was able to gather some debugging output on the issue in a job I ran last night, and so can finally file a bug report. The job I ran last night was on a 16-node cluster (all of them AWS EC2 cc2.8xlarge machines, running Ubuntu 13.04 and Cloudera CDH4.3.0). The job was a Hadoop streaming job, which reads through a large number (i.e., ~55,000) of files on S3, each of them approximately 300K bytes in size. All of the files contain 46 columns of data in each record. But I added in an extra check in my mapper code to count and verify the number of columns in every record - throwing an error and crashing the map task if the column count is wrong. If you look in the attached task logs, you'll see 2 attempts on the same task. The first one fails due to data truncated (i.e., my job intentionally fails the map task due to the current record failing the column count check). The task then gets retried on a different machine and runs to a succesful completion. You can see further evidence of the truncation further down in the task logs, where it displays the count of the records read: the failed task says 32953 records read, while the successful task says 63133. Any idea what the problem might be here and/or how to work around it? This issue is a very common occurrence on our clusters. E.g., in the job I ran last night before I had gone to bed I had already encountered 8 such failuers, and the job was only 10% complete. (~25,000 out of ~250,000 tasks.) I realize that it's common for I/O errors to occur - possibly even frequently - in a large Hadoop job. But I would think that if an I/O failure (like a truncated read) did occur, that something in the underlying infrastructure code (i.e., either in NativeS3FileSystem or in jets3t) should detect the error and throw an IOException accordingly. It shouldn't be up to the calling code to detect such failures, IMO. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10015) UserGroupInformation prints out excessive ERROR warnings
[
https://issues.apache.org/jira/browse/HADOOP-10015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790559#comment-13790559
]
Daryn Sharp commented on HADOOP-10015:
--
I guess I'm surprised that distcp is operating within an explicit doAs.
Logging doesn't occur when an exception generated under the login/current user,
just an explicit doAs.
I do agree that the logging can be annoying at times, but it's a lifesaver at
other times. Like code that loops w/o logging the exception that triggered a
retry, code that swallows an exception, code that catches and throws a generic
something failed exception, etc.
If it's a debug level, then the logging becomes worthless - it's not possible
to retroactively enable debug logging to analyze race conditions or transient
errors. Debug is too voluminous to enable for catching problems in production.
Perhaps a middle of the road solution is to create a new logger instance for
doas, named with a .doAs suffix to the ugi class. That way those that want
to disable the logging may do so via log4j.properties.
UserGroupInformation prints out excessive ERROR warnings
Key: HADOOP-10015
URL: https://issues.apache.org/jira/browse/HADOOP-10015
Project: Hadoop Common
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HADOOP-10015.000.patch, HADOOP-10015.001.patch,
HADOOP-10015.002.patch
In UserGroupInformation::doAs(), it prints out a log at ERROR level whenever
it catches an exception.
However, it prints benign warnings in the following paradigm:
{noformat}
try {
ugi.doAs(new PrivilegedExceptionActionFileStatus() {
@Override
public FileStatus run() throws Exception {
return fs.getFileStatus(nonExist);
}
});
} catch (FileNotFoundException e) {
}
{noformat}
For example, FileSystem#exists() follows this paradigm. Distcp uses this
paradigm too. The exception is expected therefore there should be no ERROR
logs printed in the namenode logs.
Currently, the user quickly finds out that the namenode log is quickly filled
by _benign_ ERROR logs when he or she runs distcp in secure set up. This
behavior confuses the operators.
This jira proposes to move the log to DEBUG level.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-10034) optimize same-filesystem symlinks by doing resolution server-side
[ https://issues.apache.org/jira/browse/HADOOP-10034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790572#comment-13790572 ] Daryn Sharp commented on HADOOP-10034: -- I don't like the performance ramifications, but I'm -1 on this because I don't see how this can work with chroot and viewfs. Symlinks must be resolved relative to the client's filesystem, not on the server itself. optimize same-filesystem symlinks by doing resolution server-side - Key: HADOOP-10034 URL: https://issues.apache.org/jira/browse/HADOOP-10034 Project: Hadoop Common Issue Type: Sub-task Components: fs Reporter: Colin Patrick McCabe We should optimize same-filesystem symlinks by doing resolution server-side rather than client side, as discussed on HADOOP-9780. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10036) RetryInvocationHandler should recognise that there is no point retrying to auth failures
[
https://issues.apache.org/jira/browse/HADOOP-10036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790586#comment-13790586
]
Daryn Sharp commented on HADOOP-10036:
--
I've seen this too, but haven't had a chance to dig into it. My initial
thought was only retry on SaslExceptions.
RetryInvocationHandler should recognise that there is no point retrying to
auth failures
Key: HADOOP-10036
URL: https://issues.apache.org/jira/browse/HADOOP-10036
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Affects Versions: 2.2.1
Reporter: Steve Loughran
Priority: Minor
The {{RetryInvocationHandler}} tries to retry connections, so as to handle
transient failures.
However, auth failures aren't treated as special, so it spins even though the
operation will not succeed with the current configuration
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-10034) optimize same-filesystem symlinks by doing resolution server-side
[ https://issues.apache.org/jira/browse/HADOOP-10034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790618#comment-13790618 ] Chris Nauroth commented on HADOOP-10034: Can we explore a hybrid approach? Suppose the client passes a representation of its viewfs mount table in NN RPC requests. Then, the NN would know that it cannot do server-side resolution if a symlink target jumps out to a different mount point. The RPC responses would need some kind of flag or optional field per path to indicate whether or not resolution was already done. Thus, all symlinks within a single mount point get resolved server-side, but we don't lose correctness on symlink targets outside the mount point. This is still likely to cut down RPC chatter in common cases. If no viewfs is in use on the client side at all, then we can represent this in the RPC requests as a degenerate case. It's a mount table with a single entry: / - hdfs://. This effectively means that the NN is free to resolve all symlink targets, because the client has no intention of resolving them differently. The downside is that we'd be leaking the viewfs abstraction down to a specific filesystem implementation. It's also a lot of protocol change, and I haven't yet considered backwards-compatibility. optimize same-filesystem symlinks by doing resolution server-side - Key: HADOOP-10034 URL: https://issues.apache.org/jira/browse/HADOOP-10034 Project: Hadoop Common Issue Type: Sub-task Components: fs Reporter: Colin Patrick McCabe We should optimize same-filesystem symlinks by doing resolution server-side rather than client side, as discussed on HADOOP-9780. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10034) optimize same-filesystem symlinks by doing resolution server-side
[
https://issues.apache.org/jira/browse/HADOOP-10034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790642#comment-13790642
]
Daryn Sharp commented on HADOOP-10034:
--
I'm of course open to possibilities but I can't see how to do a correct
solution. The suggestion here is akin to a NFS server resolving symlinks for
the client.
The leaking of abstractions into the NN is a bad idea in general. Path
resolution becomes much more complicated. Plus you're also assuming the
semantics of the client fs is a plain/vanilla viewfs. What if I'm using a
merge/union mount (not implemented, but intended per the viewfs code)? What if
I'm using a filter fs that would normally do some action based on the path?
Ie. filter it out path, rewrite it to something else, etc?
For instance, I've considered implementing a filter fs to provide transparent
har support. It would rely on seeing the har extension in the path, mask it
to look like a standard directory, and delegating the remainder of the path to
har. Symlinks in the path to the har will break this implementation because
the client won't see the har extension and the NN will throw a FNF.
Server-side resolution becomes very limiting. A better approach may be to
consider providing something like a {{DirectoryWalker}} class that can cache
the intermediate resolutions.
optimize same-filesystem symlinks by doing resolution server-side
-
Key: HADOOP-10034
URL: https://issues.apache.org/jira/browse/HADOOP-10034
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Reporter: Colin Patrick McCabe
We should optimize same-filesystem symlinks by doing resolution server-side
rather than client side, as discussed on HADOOP-9780.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[
https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790714#comment-13790714
]
Sanjay Radia commented on HADOOP-9631:
--
Here are two early comments (haven't finished reviewing the whole patch).
* viewfs#getServerDefaults(path) can be simplified. See how open or list are
implemented and it take advantage of the internal class InternalDirOfViewOf.
Something like this should work:
{code}
viewfs#getServerDefaults(f) {
InodeTree.ResolveResultAbstractFileSystem res =
fsState.resolve(getUriPath(f), true);
return res.targetFileSystem.getServerDefailts(res.remainingPath);
}
InternalDirOfViewFs#getServerDefaults() {
return LocalConfigKeys.getServerDefaults();
}
InternalDirOfViewFs#getServerDefaults(f) {
checkPathIsSlash(f);
return LocalConfigKeys.getServerDefaults();
}
{code}
* FIleSystem#getServerDefaults(f) is incorrect due to getDefaultReplication().
It should use getDefaultReplciation(f). Hence move the code for
FIleSystem#getServerDefaults() to FIleSystem#getServerDefaults(f), changing the
getDefaultReplication to pass the pathname f. Have
FIleSystem#getServerDefaults() call FIleSystem#getServerDefaults(/);
ViewFs should use underlying FileSystem's server side defaults
--
Key: HADOOP-9631
URL: https://issues.apache.org/jira/browse/HADOOP-9631
Project: Hadoop Common
Issue Type: Bug
Components: fs, viewfs
Affects Versions: 2.0.4-alpha
Reporter: Lohit Vijayarenu
Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch,
HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java
On a cluster with ViewFS as default FileSystem, creating files using
FileContext will always result with replication factor of 1, instead of
underlying filesystem default (like HDFS)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-10034) optimize same-filesystem symlinks by doing resolution server-side
[
https://issues.apache.org/jira/browse/HADOOP-10034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790727#comment-13790727
]
Colin Patrick McCabe commented on HADOOP-10034:
---
bq. The suggestion here is akin to a NFS server resolving symlinks for the
client.
With NFS, the client caches metadata for a configurable amount of time. NFS
also uses file handles extensively rather than path names. HDFS has no such
client-side caching, and we use full paths all over the place. We have nothing
to prevent the client from hammering the server when symlinks are in use.
bq. For instance, I've considered implementing a filter fs to provide
transparent har support. It would rely on seeing the har extension in the
path, mask it to look like a standard directory, and delegating the remainder
of the path to har. Symlinks in the path to the har will break this
implementation because the client won't see the har extension and the NN will
throw a FNF.
If the {{NameNode}}'s FNF contained the path it was looking for, you could
simply catch the exception, examine the path, and continue the resolution.
optimize same-filesystem symlinks by doing resolution server-side
-
Key: HADOOP-10034
URL: https://issues.apache.org/jira/browse/HADOOP-10034
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Reporter: Colin Patrick McCabe
We should optimize same-filesystem symlinks by doing resolution server-side
rather than client side, as discussed on HADOOP-9780.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HADOOP-10029) Specifying har file to MR job fails in secure cluster
[ https://issues.apache.org/jira/browse/HADOOP-10029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790862#comment-13790862 ] Sanjay Radia commented on HADOOP-10029: --- * resolvePath(). Not sure what is correct here. Resolve is suppose to follow though symlinks/mount points and resolve the path. One possibility is to make it the same as the default implementation of FileSystem (it calls fileStatus.getPath)? * Add comment why getCanonicalUri calls the underlying file system (due to tokens). * I would make ALL the copyFromXX and moveFromXX variants throw the exception rather than rely on the fact that FileSystem's default implementation calls one of copyFromXX that HarFileSystem implements and throws exception. Specifying har file to MR job fails in secure cluster - Key: HADOOP-10029 URL: https://issues.apache.org/jira/browse/HADOOP-10029 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 2.2.0 Attachments: HADOOP-10029.1.patch, HADOOP-10029.2.patch, HADOOP-10029.patch This is an issue found by [~rramya]. See the exception stack trace in the following comment. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-10031) FsShell -get/copyToLocal/moveFromLocal should support Windows local path
[ https://issues.apache.org/jira/browse/HADOOP-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10031: --- Component/s: fs Target Version/s: 3.0.0, 2.2.1 Hadoop Flags: Reviewed +1 for the patch. I verified on Mac and Windows. I plan to commit this later today. FsShell -get/copyToLocal/moveFromLocal should support Windows local path Key: HADOOP-10031 URL: https://issues.apache.org/jira/browse/HADOOP-10031 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Chuan Liu Assignee: Chuan Liu Attachments: HADOOP-10031.1.patch, HADOOP-10031.patch In current trunk code, running 'hadoop fs -get /hdfs/path c:\windows\path' will lead to an error put: unexpected URISyntaxException. This is a regression to the 1.0 fs shell commands. FsShell get, copyToLocal, and moveFromLocal should support Windows local path formats as localSrc argument to the two commands. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10031) FsShell -get/copyToLocal/moveFromLocal should support Windows local path
[ https://issues.apache.org/jira/browse/HADOOP-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790971#comment-13790971 ] Hudson commented on HADOOP-10031: - SUCCESS: Integrated in Hadoop-trunk-Commit #4577 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4577/]) HADOOP-10031. FsShell -get/copyToLocal/moveFromLocal should support Windows local path. Contributed by Chuan Liu. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1530823) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsShellCopy.java FsShell -get/copyToLocal/moveFromLocal should support Windows local path Key: HADOOP-10031 URL: https://issues.apache.org/jira/browse/HADOOP-10031 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Chuan Liu Assignee: Chuan Liu Attachments: HADOOP-10031.1.patch, HADOOP-10031.patch In current trunk code, running 'hadoop fs -get /hdfs/path c:\windows\path' will lead to an error put: unexpected URISyntaxException. This is a regression to the 1.0 fs shell commands. FsShell get, copyToLocal, and moveFromLocal should support Windows local path formats as localSrc argument to the two commands. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790972#comment-13790972 ] Sanjay Radia commented on HADOOP-9984: -- bq. Daryn, the discussion about resolved paths versus unresolved ones belongs on HADOOP-9780, not here. At least some of the points in Daryn's comments on Oct 4th apply to HADOOP-9984 , rather than HADOOP-9780. Hadoop-9984's latest patch resolves the symlinks for listStatus, i.e. if the target directory denoted by the path has children that are symlinks those symlinks will be resolved (so as to allow old apps that did if (! stat.isDir() then AssumeItIsAFile to work unchanged) Lets consider the following example: Lets say the directory /foo/bar has children a, b, c, d and lets say c is a symlink to /x/a. The method listStatus(/foo/bar) will, with the patch, return an array of FileStatus for a, b *a*, d. The repeated a is because /foo/bar/c is resolved and its target /x/a is returned. This is a spec violation: The result of listStatus is suppose to return a set of unique directory entries (since a dir cannot have duplicate names) Further if someone was using listStatus to copy the contents of /foo/bar the copy operation will fail with a FileAlreadyExistsException. Daryn gives an example of where someone is trying to do rename and gets tripped by the duplicate entry. One could argue that for some of the other issues that Daryn raises, the application writer should have been using another API. I picked the duplicates one because it breaks a fundamental invariant of a directory - ie all its children have unique names. I am not offering any solution in this comment (although I have 2 suggestions). I want us to first agree that the current patch which resolves symlinks for listStatus has a serious issue. FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default -- Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch, HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch, HADOOP-9984.012.patch, HADOOP-9984.013.patch, HADOOP-9984.014.patch, HADOOP-9984.015.patch During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-10031) FsShell -get/copyToLocal/moveFromLocal should support Windows local path
[ https://issues.apache.org/jira/browse/HADOOP-10031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10031: --- Affects Version/s: 2.2.0 3.0.0 Fix Version/s: 2.2.1 3.0.0 I have committed this to trunk, branch-2, and branch-2.2. Thank you to Chuan for contributing the patch. FsShell -get/copyToLocal/moveFromLocal should support Windows local path Key: HADOOP-10031 URL: https://issues.apache.org/jira/browse/HADOOP-10031 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 3.0.0, 2.2.0 Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 3.0.0, 2.2.1 Attachments: HADOOP-10031.1.patch, HADOOP-10031.patch In current trunk code, running 'hadoop fs -get /hdfs/path c:\windows\path' will lead to an error put: unexpected URISyntaxException. This is a regression to the 1.0 fs shell commands. FsShell get, copyToLocal, and moveFromLocal should support Windows local path formats as localSrc argument to the two commands. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10034) optimize same-filesystem symlinks by doing resolution server-side
[ https://issues.apache.org/jira/browse/HADOOP-10034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13791005#comment-13791005 ] Allen Wittenauer commented on HADOOP-10034: --- Won't doing this preclude us ever adding real relative paths into HDFS? i.e., supporting .. optimize same-filesystem symlinks by doing resolution server-side - Key: HADOOP-10034 URL: https://issues.apache.org/jira/browse/HADOOP-10034 Project: Hadoop Common Issue Type: Sub-task Components: fs Reporter: Colin Patrick McCabe We should optimize same-filesystem symlinks by doing resolution server-side rather than client side, as discussed on HADOOP-9780. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HADOOP-10038) Improve JobTracker web UI
David Chen created HADOOP-10038: --- Summary: Improve JobTracker web UI Key: HADOOP-10038 URL: https://issues.apache.org/jira/browse/HADOOP-10038 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.2.2 Reporter: David Chen Users will often need to use the JobTracker web UI to debug or tune their jobs in addition to checking the status of their jobs. The current web UI is cumbersome to navigate. The goal is to make the JobTracker web UI easier to navigate and present the data in a cleaner and more intuitive format. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-10038) Improve JobTracker web UI
[ https://issues.apache.org/jira/browse/HADOOP-10038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Chen updated HADOOP-10038: Attachment: jobtasks.png jobdetails.png jobtracker.png I have started re-styling many of the JobTracker webapp pages using Twitter Bootstrap (Flatly theme). While doing so, I realized that it would be better to start moving away from JSP and towards adding a REST API and using client-side templates and using AJAX to pull the data from the REST API. This is the same approach that is being discussed in HDFS-5333. I think we should coordinate our efforts . I know that HADOOP-7532 has been opened for the revamping the web UI for Hadoop 2. My changes here are specific for Hadoop 1.x. Improve JobTracker web UI - Key: HADOOP-10038 URL: https://issues.apache.org/jira/browse/HADOOP-10038 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.2.2 Reporter: David Chen Attachments: jobdetails.png, jobtasks.png, jobtracker.png Users will often need to use the JobTracker web UI to debug or tune their jobs in addition to checking the status of their jobs. The current web UI is cumbersome to navigate. The goal is to make the JobTracker web UI easier to navigate and present the data in a cleaner and more intuitive format. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HADOOP-10029) Specifying har file to MR job fails in secure cluster
[ https://issues.apache.org/jira/browse/HADOOP-10029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-10029: - Attachment: HADOOP-10029.3.patch bq. I would make ALL the copyFromXX and moveFromXX variants throw the exception rather than rely on the fact that FileSystem's default implementation calls one of copyFromXX that HarFileSystem implements and throws exception. There are many other methods that rely on this such as create(), createNonRecursive() etc. We will be adding all these implementations. At some point in time, we should make the internal method used in FileSystem implementation of xyz() as xyzInternal() to establish this pattern. bq. resolvePath(). Not sure what is correct here I am just restoring the previous behavior here (which was changed by HADOOP-10003). I will create a separate jira to discuss and address this. Other comments taken care of. Specifying har file to MR job fails in secure cluster - Key: HADOOP-10029 URL: https://issues.apache.org/jira/browse/HADOOP-10029 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 2.2.0 Attachments: HADOOP-10029.1.patch, HADOOP-10029.2.patch, HADOOP-10029.3.patch, HADOOP-10029.patch This is an issue found by [~rramya]. See the exception stack trace in the following comment. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-10034) optimize same-filesystem symlinks by doing resolution server-side
[
https://issues.apache.org/jira/browse/HADOOP-10034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13791048#comment-13791048
]
Colin Patrick McCabe commented on HADOOP-10034:
---
bq. Won't doing this preclude us ever adding real relative paths into HDFS?
i.e., supporting ..
I assume you mean symlinks to {{..}}. We do have those already-- hence the
discussion about making the client's mount table accessible to the server.
Tangent: I have to admit, I'm a little confused by the idea of relative
symlinks that include a scheme other than null. Are we going to allow those,
and if so, what should the behavior be?
optimize same-filesystem symlinks by doing resolution server-side
-
Key: HADOOP-10034
URL: https://issues.apache.org/jira/browse/HADOOP-10034
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Reporter: Colin Patrick McCabe
We should optimize same-filesystem symlinks by doing resolution server-side
rather than client side, as discussed on HADOOP-9780.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HADOOP-9835) Identity Token Service API
[ https://issues.apache.org/jira/browse/HADOOP-9835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-9835: --- Issue Type: Sub-task (was: Task) Parent: HADOOP-9392 Identity Token Service API -- Key: HADOOP-9835 URL: https://issues.apache.org/jira/browse/HADOOP-9835 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 3.0.0 Reporter: Yi Liu Labels: Rhino Identity token service is used by client to request identity token with authentication result after client authenticates with authentication service. This JIRA is to define Identity token service API, and the pluggable framework allows different implementation. The scope of this task is highlighted as following: • Define Identity token service API. • Specify how to configure and register Identity token service implementation. -- This message was sent by Atlassian JIRA (v6.1#6144)
