[jira] [Updated] (HADOOP-9762) RetryCache utility for implementing RPC retries
[ https://issues.apache.org/jira/browse/HADOOP-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9762: Attachment: HADOOP-9762.3.patch Changed from byte[] for clientId to two longs to save memory related to the cost of Array. RetryCache utility for implementing RPC retries --- Key: HADOOP-9762 URL: https://issues.apache.org/jira/browse/HADOOP-9762 Project: Hadoop Common Issue Type: Improvement Components: util Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HADOOP-9762.1.patch, HADOOP-9762.2.patch, HADOOP-9762.3.patch, HADOOP-9762.patch HDFS-4979 has the RetryCache implementation in uncommitted patches. This jira moves this utility to hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9761) ViewFileSystem#rename fails when using DistributedFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-9761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-9761: Status: Patch Available (was: Open) ViewFileSystem#rename fails when using DistributedFileSystem Key: HADOOP-9761 URL: https://issues.apache.org/jira/browse/HADOOP-9761 Project: Hadoop Common Issue Type: Bug Components: viewfs Affects Versions: 3.0.0, 2.1.0-beta Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hadoop-9761-1.patch ViewFileSystem currently passes unqualified paths (no scheme or authority) to underlying FileSystems when doing a rename. DistributedFileSystem symlink support added in HADOOP-9418 needs to qualify and check rename sources and destinations since cross-filesystem renames aren't supported, so this breaks in the following way - Default FS URI is configured to viewfs://viewfs - When doing a rename, ViewFileSystem checks to make sure both src and dst FileSystems are the same (which they are, both in same DFS), and then calls DistributedFileSystem#rename with unqualified remainder paths - Since these paths are unqualified, DFS qualifies them with the default FS to check that it can do the rename. This turns it into viewfs://viewfs/path - Since viewfs://viewfs is not the DFS's URI, DFS errors out the rename. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9761) ViewFileSystem#rename fails when using DistributedFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-9761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-9761: Attachment: hadoop-9761-1.patch Had to think this one over a bit. FileSystem isn't like FileContext, in that given an unqualified path, it doesn't try to qualify it against the default FS first. It's supposed to just treat the path as one of its own. So, simply move the cross-FS check in DFS#rename. Also tweak FileSystemLinkResolver to match. The existing test cases weren't picking this up because the DFS returned from the minicluster use the minicluster conf, rather than the viewfs client's conf like in a real environment. So, I make sure to set the DFS defaultFS to the viewfs. I tested fail before/works after with the testcase and tested with the shell. Also took the opportunity to correct some javadoc in DFS. ViewFileSystem#rename fails when using DistributedFileSystem Key: HADOOP-9761 URL: https://issues.apache.org/jira/browse/HADOOP-9761 Project: Hadoop Common Issue Type: Bug Components: viewfs Affects Versions: 3.0.0, 2.1.0-beta Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hadoop-9761-1.patch ViewFileSystem currently passes unqualified paths (no scheme or authority) to underlying FileSystems when doing a rename. DistributedFileSystem symlink support added in HADOOP-9418 needs to qualify and check rename sources and destinations since cross-filesystem renames aren't supported, so this breaks in the following way - Default FS URI is configured to viewfs://viewfs - When doing a rename, ViewFileSystem checks to make sure both src and dst FileSystems are the same (which they are, both in same DFS), and then calls DistributedFileSystem#rename with unqualified remainder paths - Since these paths are unqualified, DFS qualifies them with the default FS to check that it can do the rename. This turns it into viewfs://viewfs/path - Since viewfs://viewfs is not the DFS's URI, DFS errors out the rename. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9762) RetryCache utility for implementing RPC retries
[ https://issues.apache.org/jira/browse/HADOOP-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-9762: -- Status: Patch Available (was: Open) RetryCache utility for implementing RPC retries --- Key: HADOOP-9762 URL: https://issues.apache.org/jira/browse/HADOOP-9762 Project: Hadoop Common Issue Type: Improvement Components: util Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HADOOP-9762.1.patch, HADOOP-9762.2.patch, HADOOP-9762.3.patch, HADOOP-9762.patch HDFS-4979 has the RetryCache implementation in uncommitted patches. This jira moves this utility to hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9763) Extends LightWeightGSet to support eviction of expired elements
[ https://issues.apache.org/jira/browse/HADOOP-9763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-9763: -- Resolution: Fixed Fix Version/s: 2.1.0-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to trunk, branch-2 and branch-2.1-beta. Extends LightWeightGSet to support eviction of expired elements --- Key: HADOOP-9763 URL: https://issues.apache.org/jira/browse/HADOOP-9763 Project: Hadoop Common Issue Type: New Feature Components: util Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 2.1.0-beta Attachments: c9763_20130723b.patch, c9763_20130723.patch, c9763_20130724.patch, h5017_20130721.patch, h5017_20130722b.patch, h5017_20130722.patch We extends LightWeightGSet to support eviction of expired elements so that namenode could use it as the RetryCache. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9762) RetryCache utility for implementing RPC retries
[
https://issues.apache.org/jira/browse/HADOOP-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718080#comment-13718080
]
Hadoop QA commented on HADOOP-9762:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12593859/HADOOP-9762.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2835//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2835//console
This message is automatically generated.
RetryCache utility for implementing RPC retries
---
Key: HADOOP-9762
URL: https://issues.apache.org/jira/browse/HADOOP-9762
Project: Hadoop Common
Issue Type: Improvement
Components: util
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
Attachments: HADOOP-9762.1.patch, HADOOP-9762.2.patch,
HADOOP-9762.3.patch, HADOOP-9762.patch
HDFS-4979 has the RetryCache implementation in uncommitted patches. This jira
moves this utility to hadoop-common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9762) RetryCache utility for implementing RPC retries
[ https://issues.apache.org/jira/browse/HADOOP-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718086#comment-13718086 ] Jing Zhao commented on HADOOP-9762: --- +1 for the new patch. RetryCache utility for implementing RPC retries --- Key: HADOOP-9762 URL: https://issues.apache.org/jira/browse/HADOOP-9762 Project: Hadoop Common Issue Type: Improvement Components: util Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HADOOP-9762.1.patch, HADOOP-9762.2.patch, HADOOP-9762.3.patch, HADOOP-9762.patch HDFS-4979 has the RetryCache implementation in uncommitted patches. This jira moves this utility to hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9392) Token based authentication and Single Sign On
[ https://issues.apache.org/jira/browse/HADOOP-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718088#comment-13718088 ] Kai Zheng commented on HADOOP-9392: --- Thanks for your thoughts. Sorry for being late to reply this as we're busy with breaking down TokenAuth and working on our initial patch. HAS (Hadoop A* Server) is good and I like it. The 'A' of HAS could be explained as Authentication, Authorization, or Auditing or more of them, depending on HAS is provisioned with which role(s). In this way it's much flexible and better to evolve in future. In high level considerations, we may need centralized Authentication Server, centralized Authorization Server, or even centralized Auditing Server, and such servers would be great to be combined into one centralized server, or deployed separately regarding network/multi-tenancy concerns. Currently we're mainly focusing on Authentication and Authorization aspects, and the two roles can be provisioned into one server, or separate servers, and the server can be just unified and named as HAS. In TokenAuth, we use Identity Token and Access Token in places where appropriate, and only mention token in contexts that the token can be clearly interpreted as either Identity Token or Access Token. If HAS is accepted, then we can use “HAS tokens” to mean Identity/Access Tokens in TokenAuth as you suggested. Considering Hadoop existing tokens such as delegation token/block token/job token and the like are widely used today we can continue to use “Hadoop tokens” to mean them. Please let me know what's your further thought about this. Thanks. Token based authentication and Single Sign On - Key: HADOOP-9392 URL: https://issues.apache.org/jira/browse/HADOOP-9392 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Kai Zheng Assignee: Kai Zheng Fix For: 3.0.0 Attachments: token-based-authn-plus-sso.pdf, token-based-authn-plus-sso-v2.0.pdf This is an umbrella entry for one of project Rhino’s topic, for details of project Rhino, please refer to https://github.com/intel-hadoop/project-rhino/. The major goal for this entry as described in project Rhino was “Core, HDFS, ZooKeeper, and HBase currently support Kerberos authentication at the RPC layer, via SASL. However this does not provide valuable attributes such as group membership, classification level, organizational identity, or support for user defined attributes. Hadoop components must interrogate external resources for discovering these attributes and at scale this is problematic. There is also no consistent delegation model. HDFS has a simple delegation capability, and only Oozie can take limited advantage of it. We will implement a common token based authentication framework to decouple internal user and service authentication from external mechanisms used to support it (like Kerberos)” We’d like to start our work from Hadoop-Common and try to provide common facilities by extending existing authentication framework which support: 1.Pluggable token provider interface 2.Pluggable token verification protocol and interface 3.Security mechanism to distribute secrets in cluster nodes 4.Delegation model of user authentication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9762) RetryCache utility for implementing RPC retries
[ https://issues.apache.org/jira/browse/HADOOP-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-9762: -- Resolution: Fixed Fix Version/s: 2.1.0-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to trunk, branch-2 and branch-2.1-beta. RetryCache utility for implementing RPC retries --- Key: HADOOP-9762 URL: https://issues.apache.org/jira/browse/HADOOP-9762 Project: Hadoop Common Issue Type: Improvement Components: util Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 2.1.0-beta Attachments: HADOOP-9762.1.patch, HADOOP-9762.2.patch, HADOOP-9762.3.patch, HADOOP-9762.patch HDFS-4979 has the RetryCache implementation in uncommitted patches. This jira moves this utility to hadoop-common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9706) Provide Hadoop Karaf support
[ https://issues.apache.org/jira/browse/HADOOP-9706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718339#comment-13718339 ] Thierry Caminel commented on HADOOP-9706: - Excellent initiative. I hability to deploy MR jobs in Karaf, export Hadoop services as OSGi services and thus inheriting the full power of OSGi might ease a lot the integration of Hadoop with ESB. Continue! Provide Hadoop Karaf support Key: HADOOP-9706 URL: https://issues.apache.org/jira/browse/HADOOP-9706 Project: Hadoop Common Issue Type: Task Components: tools Reporter: Jean-Baptiste Onofré Fix For: 3.0.0 Attachments: HADOOP-9706.patch, Karaf-HDFS-client.pdf To follow the discussion about OSGi, and in order to move forward, I propose the following hadoop-karaf bundle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9698) RPCv9 client must honor server's SASL negotiate response
[ https://issues.apache.org/jira/browse/HADOOP-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718380#comment-13718380 ] Daryn Sharp commented on HADOOP-9698: - I believe Sanjay is actively reviewing. RPCv9 client must honor server's SASL negotiate response Key: HADOOP-9698 URL: https://issues.apache.org/jira/browse/HADOOP-9698 Project: Hadoop Common Issue Type: Sub-task Components: ipc Affects Versions: 3.0.0, 2.1.0-beta Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-9698.patch As of HADOOP-9421, a RPCv9 server will advertise its authentication methods. This is meant to support features such as IP failover, better token selection, and interoperability in a heterogenous security environment. Currently the client ignores the negotiate response and just blindly attempts to authenticate instead of choosing a mutually agreeable auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9238) FsShell -put from stdin auto-creates paths
[ https://issues.apache.org/jira/browse/HADOOP-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718488#comment-13718488 ] Daryn Sharp commented on HADOOP-9238: - Did you run the hdfs FsShell tests like TestHDFSCLI? Also, please add a test case to it. FsShell -put from stdin auto-creates paths -- Key: HADOOP-9238 URL: https://issues.apache.org/jira/browse/HADOOP-9238 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Attachments: HADOOP-9238.patch FsShell put is no longer supposed to auto-create paths. There's an inconsistency where a put from stdin will still auto-create paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9755) HADOOP-9164 breaks the windows native build
[
https://issues.apache.org/jira/browse/HADOOP-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718544#comment-13718544
]
Luke Lu commented on HADOOP-9755:
-
HADOOP-9759 is resolved/fixed first :)
HADOOP-9164 breaks the windows native build
---
Key: HADOOP-9755
URL: https://issues.apache.org/jira/browse/HADOOP-9755
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Vinay
Assignee: Binglin Chang
Priority: Blocker
Attachments: HADOOP-9755.patch, HADOOP-9755.v2.patch
After HADOOP-9164 hadooop windows native build is broken.
{noformat} NativeCodeLoader.c
src\org\apache\hadoop\util\NativeCodeLoader.c(41): error C2065: 'Dl_info' :
undeclared identifier
[D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
src\org\apache\hadoop\util\NativeCodeLoader.c(41): error C2146: syntax error
: missing ';' before identifier 'dl_info'
[D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
src\org\apache\hadoop\util\NativeCodeLoader.c(41): error C2065: 'dl_info' :
undeclared identifier
[D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
src\org\apache\hadoop\util\NativeCodeLoader.c(42): error C2143: syntax error
: missing ';' before 'type'
[D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
src\org\apache\hadoop\util\NativeCodeLoader.c(45): error C2065: 'ret' :
undeclared identifier
[D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
src\org\apache\hadoop\util\NativeCodeLoader.c(45): error C2065: 'dl_info' :
undeclared identifier
[D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
src\org\apache\hadoop\util\NativeCodeLoader.c(45): error C2224: left of
'.dli_fname' must have struct/union type
[D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
src\org\apache\hadoop\util\NativeCodeLoader.c(45): fatal error C1903: unable
to recover from previous error(s); stopping compilation
[D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
NativeCrc32.c
Done Building Project
D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj
(default targets) -- FAILED.
Done Building Project
D:\hdp2\hadoop-common-project\hadoop-common\src\main\native\native.sln
(default targets) -- FAILED.{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9698) RPCv9 client must honor server's SASL negotiate response
[
https://issues.apache.org/jira/browse/HADOOP-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718693#comment-13718693
]
Jitendra Nath Pandey commented on HADOOP-9698:
--
- It seems from the code that client always sends NEGOTIATE first for sasl
case. IMO, the client should initiate with its preferred auth type and server
should negotiate only if it doesn't support/allow the requested auth type,
otherwise we negotiation would happen unnecessarily for most common case where
both support the same auth type.
- It is not clear to me why client cannot figure out whether it has the token
for the service. The client should try sasl only if it has tokens for the
service in question. I
t may be possible that the ugi has tokens for one service and not for others
but security is disabled.
- Do we always allow fallback if the client wants token with security disabled
but server says simple?
{code}
+} else if (UserGroupInformation.isSecurityEnabled()
+ !fallbackAllowed) {
{code}
- I would recommend to avoid passing configuration object to SaslRpcClient and
storing it there.
RPCv9 client must honor server's SASL negotiate response
Key: HADOOP-9698
URL: https://issues.apache.org/jira/browse/HADOOP-9698
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Blocker
Attachments: HADOOP-9698.patch
As of HADOOP-9421, a RPCv9 server will advertise its authentication methods.
This is meant to support features such as IP failover, better token
selection, and interoperability in a heterogenous security environment.
Currently the client ignores the negotiate response and just blindly attempts
to authenticate instead of choosing a mutually agreeable auth method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9767) RawLocalFileSystem#rename does not correctly enforce replacement of empty destination directory on all platforms
Chris Nauroth created HADOOP-9767:
-
Summary: RawLocalFileSystem#rename does not correctly enforce
replacement of empty destination directory on all platforms
Key: HADOOP-9767
URL: https://issues.apache.org/jira/browse/HADOOP-9767
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
POSIX {{rename}} semantics state that renaming a source directory to an
existing destination directory that is empty will act as a full replacement of
that existing destination. {{RawLocalFileSystem#rename}} calls Java's
{{File#renameTo}}, which provides these semantics on most platforms. On some
platforms (notably Windows), {{File#renameTo}} does not provide this behavior.
This causes problems for MapReduce and other downstream components like Hive,
HCatalog, and Sqoop that depend on the replacement behavior.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718809#comment-13718809
]
Chris Nauroth commented on HADOOP-9507:
---
I'm about to post some patches that resolve this and also some other cases.
I'm pasting a comment I originally entered on HADOOP-9767 that explains the
root cause:
POSIX rename semantics state that renaming a source directory to an existing
destination directory that is empty will act as a full replacement of that
existing destination. RawLocalFileSystem#rename calls Java's File#renameTo,
which provides these semantics on most platforms. On some platforms (notably
Windows), File#renameTo does not provide this behavior. This causes problems
for MapReduce and other downstream components like Hive, HCatalog, and Sqoop
that depend on the replacement behavior.
LocalFileSystem rename() is broken in some cases when destination exists
Key: HADOOP-9507
URL: https://issues.apache.org/jira/browse/HADOOP-9507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Mostafa Elhemali
Assignee: Daryn Sharp
Priority: Minor
Attachments: HADOOP-9507.branch-1-win.patch
The rename() method in RawLocalFileSystem uses FileUtil.copy() without
realizing that FileUtil.copy() has a special behavior that if you're copying
/foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
instead of overwriting it, which is not what rename() wants. So you end up
with weird behaviors like in this repro:
{code}
c:
cd \
md Foo
md Bar
md Foo\X
md Bar\X
hadoop fs -mv file:///c:/Foo file:///c:/Bar
{code}
At the end of this, you would expect to find only Bar\X, but you instead find
Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9767) RawLocalFileSystem#rename does not correctly enforce replacement of empty destination directory on all platforms
[
https://issues.apache.org/jira/browse/HADOOP-9767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HADOOP-9767.
---
Resolution: Duplicate
I just realized that this is basically a duplicate of HADOOP-9507, so I'll post
patches over there.
RawLocalFileSystem#rename does not correctly enforce replacement of empty
destination directory on all platforms
Key: HADOOP-9767
URL: https://issues.apache.org/jira/browse/HADOOP-9767
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
POSIX {{rename}} semantics state that renaming a source directory to an
existing destination directory that is empty will act as a full replacement
of that existing destination. {{RawLocalFileSystem#rename}} calls Java's
{{File#renameTo}}, which provides these semantics on most platforms. On some
platforms (notably Windows), {{File#renameTo}} does not provide this
behavior. This causes problems for MapReduce and other downstream components
like Hive, HCatalog, and Sqoop that depend on the replacement behavior.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9507:
--
Target Version/s: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Affects Version/s: 1.3.0
2.1.0-beta
1-win
3.0.0
Assignee: Chris Nauroth (was: Daryn Sharp)
LocalFileSystem rename() is broken in some cases when destination exists
Key: HADOOP-9507
URL: https://issues.apache.org/jira/browse/HADOOP-9507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Mostafa Elhemali
Assignee: Chris Nauroth
Priority: Minor
Attachments: HADOOP-9507-branch-1-win.1.patch,
HADOOP-9507.branch-1-win.patch
The rename() method in RawLocalFileSystem uses FileUtil.copy() without
realizing that FileUtil.copy() has a special behavior that if you're copying
/foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
instead of overwriting it, which is not what rename() wants. So you end up
with weird behaviors like in this repro:
{code}
c:
cd \
md Foo
md Bar
md Foo\X
md Bar\X
hadoop fs -mv file:///c:/Foo file:///c:/Bar
{code}
At the end of this, you would expect to find only Bar\X, but you instead find
Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9507:
--
Attachment: HADOOP-9507-branch-1-win.1.patch
LocalFileSystem rename() is broken in some cases when destination exists
Key: HADOOP-9507
URL: https://issues.apache.org/jira/browse/HADOOP-9507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Mostafa Elhemali
Assignee: Chris Nauroth
Priority: Minor
Attachments: HADOOP-9507-branch-1-win.1.patch,
HADOOP-9507.branch-1-win.patch
The rename() method in RawLocalFileSystem uses FileUtil.copy() without
realizing that FileUtil.copy() has a special behavior that if you're copying
/foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
instead of overwriting it, which is not what rename() wants. So you end up
with weird behaviors like in this repro:
{code}
c:
cd \
md Foo
md Bar
md Foo\X
md Bar\X
hadoop fs -mv file:///c:/Foo file:///c:/Bar
{code}
At the end of this, you would expect to find only Bar\X, but you instead find
Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9507:
--
Attachment: HADOOP-9507-branch-1.1.patch
LocalFileSystem rename() is broken in some cases when destination exists
Key: HADOOP-9507
URL: https://issues.apache.org/jira/browse/HADOOP-9507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Mostafa Elhemali
Assignee: Chris Nauroth
Priority: Minor
Attachments: HADOOP-9507-branch-1.1.patch,
HADOOP-9507-branch-1-win.1.patch, HADOOP-9507.branch-1-win.patch
The rename() method in RawLocalFileSystem uses FileUtil.copy() without
realizing that FileUtil.copy() has a special behavior that if you're copying
/foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
instead of overwriting it, which is not what rename() wants. So you end up
with weird behaviors like in this repro:
{code}
c:
cd \
md Foo
md Bar
md Foo\X
md Bar\X
hadoop fs -mv file:///c:/Foo file:///c:/Bar
{code}
At the end of this, you would expect to find only Bar\X, but you instead find
Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9507:
--
Status: Patch Available (was: Reopened)
LocalFileSystem rename() is broken in some cases when destination exists
Key: HADOOP-9507
URL: https://issues.apache.org/jira/browse/HADOOP-9507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Mostafa Elhemali
Assignee: Chris Nauroth
Priority: Minor
Attachments: HADOOP-9507-branch-1.1.patch,
HADOOP-9507-branch-1-win.1.patch, HADOOP-9507.branch-1-win.patch,
HADOOP-9507-trunk.1.patch
The rename() method in RawLocalFileSystem uses FileUtil.copy() without
realizing that FileUtil.copy() has a special behavior that if you're copying
/foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
instead of overwriting it, which is not what rename() wants. So you end up
with weird behaviors like in this repro:
{code}
c:
cd \
md Foo
md Bar
md Foo\X
md Bar\X
hadoop fs -mv file:///c:/Foo file:///c:/Bar
{code}
At the end of this, you would expect to find only Bar\X, but you instead find
Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9507:
--
Attachment: HADOOP-9507-trunk.1.patch
I've attached patches for trunk, branch-1, and branch-1-win. The branch-1-win
patch is slightly different, because there had been a prior attempt to fix this
issue on that branch, but it wasn't quite correct.
I've successfully tested these patches on Mac, Linux, and Windows.
LocalFileSystem rename() is broken in some cases when destination exists
Key: HADOOP-9507
URL: https://issues.apache.org/jira/browse/HADOOP-9507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Mostafa Elhemali
Assignee: Chris Nauroth
Priority: Minor
Attachments: HADOOP-9507-branch-1.1.patch,
HADOOP-9507-branch-1-win.1.patch, HADOOP-9507.branch-1-win.patch,
HADOOP-9507-trunk.1.patch
The rename() method in RawLocalFileSystem uses FileUtil.copy() without
realizing that FileUtil.copy() has a special behavior that if you're copying
/foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
instead of overwriting it, which is not what rename() wants. So you end up
with weird behaviors like in this repro:
{code}
c:
cd \
md Foo
md Bar
md Foo\X
md Bar\X
hadoop fs -mv file:///c:/Foo file:///c:/Bar
{code}
At the end of this, you would expect to find only Bar\X, but you instead find
Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9698) RPCv9 client must honor server's SASL negotiate response
[ https://issues.apache.org/jira/browse/HADOOP-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718816#comment-13718816 ] Daryn Sharp commented on HADOOP-9698: - bq. It seems from the code that client always sends NEGOTIATE first for sasl case. IMO, the client should initiate with its preferred auth type and server should negotiate only if it doesn't support/allow the requested auth type The negotiate/initiate issue was hashed out in HADOOP-9421. This jira's headline is about having the client honor the negotiate. The capability for the client to blindly INITIATE is there, but it's counter to the higher level goals: IP failover where the client doesn't know the service principal, improved token selection which may change the proto/serverId. bq. It is not clear to me why client cannot figure out whether it has the token for the service. The client should try sasl only if it has tokens for the service in question The server should be able to tell the client, or at least hint to it, what token it wants via the proto/serverId fields instead of the client trying to guess. # The current guessing model doesn't work with the default use_ip=true because multi-interface hosts, NATing, public/private networks, etc will fail to find a token. # The use_ip=false case fixes some of those issue, but it's fragile to ensure client code manipulates the addr correctly. I want out of that janitorial job - something at the rpc layer prevents user error. # HA token selection with a failover proxy is convoluted and can be simplified. bq. Do we always allow fallback if the client wants token with security disabled but server says simple? The question is a bit contradictory. It's not what the client wants, but rather what the server wants. There's no way for the client to know it's expected to use a token. So yes, if a server tells a simple client to do simple, it will do simple. RPCv9 client must honor server's SASL negotiate response Key: HADOOP-9698 URL: https://issues.apache.org/jira/browse/HADOOP-9698 Project: Hadoop Common Issue Type: Sub-task Components: ipc Affects Versions: 3.0.0, 2.1.0-beta Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-9698.patch As of HADOOP-9421, a RPCv9 server will advertise its authentication methods. This is meant to support features such as IP failover, better token selection, and interoperability in a heterogenous security environment. Currently the client ignores the negotiate response and just blindly attempts to authenticate instead of choosing a mutually agreeable auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718822#comment-13718822
]
Chuan Liu commented on HADOOP-9507:
---
+1
Thanks for the detailed analysis, Chris!
I don't remember the subtle differences of File#renameTo() on Windows and Linux.
I verified the differences in renaming behavior, and agree with your fix.
LocalFileSystem rename() is broken in some cases when destination exists
Key: HADOOP-9507
URL: https://issues.apache.org/jira/browse/HADOOP-9507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Mostafa Elhemali
Assignee: Chris Nauroth
Priority: Minor
Attachments: HADOOP-9507-branch-1.1.patch,
HADOOP-9507-branch-1-win.1.patch, HADOOP-9507.branch-1-win.patch,
HADOOP-9507-trunk.1.patch
The rename() method in RawLocalFileSystem uses FileUtil.copy() without
realizing that FileUtil.copy() has a special behavior that if you're copying
/foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
instead of overwriting it, which is not what rename() wants. So you end up
with weird behaviors like in this repro:
{code}
c:
cd \
md Foo
md Bar
md Foo\X
md Bar\X
hadoop fs -mv file:///c:/Foo file:///c:/Bar
{code}
At the end of this, you would expect to find only Bar\X, but you instead find
Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718852#comment-13718852
]
Hadoop QA commented on HADOOP-9507:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594027/HADOOP-9507-trunk.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1152 javac
compiler warnings (more than the trunk's current 1151 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2838//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2838//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2838//console
This message is automatically generated.
LocalFileSystem rename() is broken in some cases when destination exists
Key: HADOOP-9507
URL: https://issues.apache.org/jira/browse/HADOOP-9507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Mostafa Elhemali
Assignee: Chris Nauroth
Priority: Minor
Attachments: HADOOP-9507-branch-1.1.patch,
HADOOP-9507-branch-1-win.1.patch, HADOOP-9507.branch-1-win.patch,
HADOOP-9507-trunk.1.patch
The rename() method in RawLocalFileSystem uses FileUtil.copy() without
realizing that FileUtil.copy() has a special behavior that if you're copying
/foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
instead of overwriting it, which is not what rename() wants. So you end up
with weird behaviors like in this repro:
{code}
c:
cd \
md Foo
md Bar
md Foo\X
md Bar\X
hadoop fs -mv file:///c:/Foo file:///c:/Bar
{code}
At the end of this, you would expect to find only Bar\X, but you instead find
Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable
Chris Nauroth created HADOOP-9768:
-
Summary: chown and chgrp reject users and groups with spaces on
platforms where spaces are otherwise acceptable
Key: HADOOP-9768
URL: https://issues.apache.org/jira/browse/HADOOP-9768
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
The chown and chgrp commands enforce a check on a valid set of characters for
user and group. The set of valid characters does not include space. On some
platforms (notably Windows), the space character is acceptable. We've seen
this cause test failures in {{TestFsShellReturnCode}} when running on Windows
if the logged-in user is a member of a group like Remote Desktop Users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9701) mvn site ambiguous links in hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718929#comment-13718929
]
Alejandro Abdelnur commented on HADOOP-9701:
+1
mvn site ambiguous links in hadoop-common
-
Key: HADOOP-9701
URL: https://issues.apache.org/jira/browse/HADOOP-9701
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Steve Loughran
Assignee: Karthik Kambatla
Priority: Minor
Attachments: hadoop-9701-1.patch
{code}
[INFO] Rendering site with org.apache.maven.skins:maven-stylus-skin:jar:1.2
skin.
[WARNING] [APT Parser] Ambiguous link: 'InterfaceClassification.html'. If
this is a local link, prepend ./!
{code}
Also, noticed a warning in SingleNodeSetup.apt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718943#comment-13718943
]
Colin Patrick McCabe commented on HADOOP-9652:
--
bq. True, I added the username check back in. The other tests are unfortunately
harder; for instance, symlink permissions are inconsistent across operating
systems, and it's unclear to me which of the user's groups is set.
Newly created files get set to the user's primary group on UNIX.
{code}
+FileStatus status = fsImpl.getFileLinkStatus(f);
+// FileSystem#getFileLinkStatus qualifies the link target
+// AbstractFileSystem needs to return it plain since it's qualified
+// in FileContext, so re-get and set the plain target
+if (status.isSymlink()) {
+ status.setSymlink(fsImpl.getLinkTarget(f));
+}
+return status;
{code}
This actually highlight something that I think we might need to fix in the
FileSystem symlink support-- the fact that it's impossible to view relative
symlink paths once they are created.
Let's say I create a symlink from a - ./b in my local HDFS.
With the new FSShell support in HDFS-4019, Hadoop displays this as:
{code}
lrwxrwxrwx - cmccabe supergroup 0 2013-07-23 15:59 /a -
hdfs://localhost:6000/b
{code}
It seems to me that we should display relative symlinks *as* relative, rather
than absolute-izing them and adding a URI. Otherwise, if we use DistCp to copy
a cluster's data to another cluster, all the symlink targets will be wrong.
This is probably out of scope for this particular JIRA, but should we file a
follow-up?
{code}
+if (osType == OSType.OS_TYPE_UNIX) {
+ return new String[] {
+ bash, -c,
+ exec 'stat' ' + derefFlag + c' '%s,%F,%Y,%X,%a,%U,%G,%N' '
+ + path + ' 21 };
{code}
Shouldn't this be OS_TYPE_LINUX? I don't think, for example, OpenBSD comes
with the GNU stat program. We should also really also file a follow-up bug to
remove this and move to a more portable solution when we get the green light to
drop JDK6 support.
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9701) mvn site ambiguous links in hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-9701:
---
Resolution: Fixed
Fix Version/s: 2.1.0-beta
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Karthik. Committed to trunk, branch-2 and branch-2.1.
mvn site ambiguous links in hadoop-common
-
Key: HADOOP-9701
URL: https://issues.apache.org/jira/browse/HADOOP-9701
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Steve Loughran
Assignee: Karthik Kambatla
Priority: Minor
Fix For: 2.1.0-beta
Attachments: hadoop-9701-1.patch
{code}
[INFO] Rendering site with org.apache.maven.skins:maven-stylus-skin:jar:1.2
skin.
[WARNING] [APT Parser] Ambiguous link: 'InterfaceClassification.html'. If
this is a local link, prepend ./!
{code}
Also, noticed a warning in SingleNodeSetup.apt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9761) ViewFileSystem#rename fails when using DistributedFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-9761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718969#comment-13718969
]
Colin Patrick McCabe commented on HADOOP-9761:
--
{code}
// Assumes path belongs to this FileSystem.
// Callers are responsible for checking this
FileSystem fs = filesys;
{code}
I don't see where you are actually checking that the path belongs to the
FileSystem in most of the callers.
For example, back before symlinks support in FileSystem, code like this:
{code}
FileSystem fs = FileSystem.get(hdfs...);
fs.open(file:///boguspath);
{code}
Would generate a stacktrace like this:
{code}
Exception in thread main java.lang.IllegalArgumentException: Wrong FS:
file:/boguspath, expected: hdfs://localhost:6000
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:590)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:170)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:246)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:79)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:711)
{code}
After the symlinks support, it generated this:
{code}
Exception in thread main java.lang.IllegalArgumentException: Wrong FS:
file:/tmp/t, expected: hdfs://localhost:6000
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:636)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:181)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:91)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:290)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:286)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:78)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:286)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:757)
{code}
It seems like this change would simply pass the incorrect path through to the
FileSystem, which seems wrong.
treating unqualified (i.e., schemeless) Paths as owned by the current
FileSystem should not require disabling the checks for Paths which *do* have a
wrong scheme.
ViewFileSystem#rename fails when using DistributedFileSystem
Key: HADOOP-9761
URL: https://issues.apache.org/jira/browse/HADOOP-9761
Project: Hadoop Common
Issue Type: Bug
Components: viewfs
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: hadoop-9761-1.patch
ViewFileSystem currently passes unqualified paths (no scheme or authority) to
underlying FileSystems when doing a rename. DistributedFileSystem symlink
support added in HADOOP-9418 needs to qualify and check rename sources and
destinations since cross-filesystem renames aren't supported, so this breaks
in the following way
- Default FS URI is configured to viewfs://viewfs
- When doing a rename, ViewFileSystem checks to make sure both src and dst
FileSystems are the same (which they are, both in same DFS), and then calls
DistributedFileSystem#rename with unqualified remainder paths
- Since these paths are unqualified, DFS qualifies them with the default FS
to check that it can do the rename. This turns it into
viewfs://viewfs/path
- Since viewfs://viewfs is not the DFS's URI, DFS errors out the rename.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718965#comment-13718965
]
Mostafa Elhemali commented on HADOOP-9507:
--
Thanks Chris - +1.
LocalFileSystem rename() is broken in some cases when destination exists
Key: HADOOP-9507
URL: https://issues.apache.org/jira/browse/HADOOP-9507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Mostafa Elhemali
Assignee: Chris Nauroth
Priority: Minor
Attachments: HADOOP-9507-branch-1.1.patch,
HADOOP-9507-branch-1-win.1.patch, HADOOP-9507.branch-1-win.patch,
HADOOP-9507-trunk.1.patch
The rename() method in RawLocalFileSystem uses FileUtil.copy() without
realizing that FileUtil.copy() has a special behavior that if you're copying
/foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
instead of overwriting it, which is not what rename() wants. So you end up
with weird behaviors like in this repro:
{code}
c:
cd \
md Foo
md Bar
md Foo\X
md Bar\X
hadoop fs -mv file:///c:/Foo file:///c:/Bar
{code}
At the end of this, you would expect to find only Bar\X, but you instead find
Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable
[
https://issues.apache.org/jira/browse/HADOOP-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9768:
--
Attachment: HADOOP-9768-branch-1.1.patch
chown and chgrp reject users and groups with spaces on platforms where spaces
are otherwise acceptable
--
Key: HADOOP-9768
URL: https://issues.apache.org/jira/browse/HADOOP-9768
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-9768-branch-1.1.patch
The chown and chgrp commands enforce a check on a valid set of characters for
user and group. The set of valid characters does not include space. On some
platforms (notably Windows), the space character is acceptable. We've seen
this cause test failures in {{TestFsShellReturnCode}} when running on Windows
if the logged-in user is a member of a group like Remote Desktop Users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable
[
https://issues.apache.org/jira/browse/HADOOP-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9768:
--
Status: Patch Available (was: Open)
chown and chgrp reject users and groups with spaces on platforms where spaces
are otherwise acceptable
--
Key: HADOOP-9768
URL: https://issues.apache.org/jira/browse/HADOOP-9768
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-9768-branch-1.1.patch, HADOOP-9768-trunk.1.patch
The chown and chgrp commands enforce a check on a valid set of characters for
user and group. The set of valid characters does not include space. On some
platforms (notably Windows), the space character is acceptable. We've seen
this cause test failures in {{TestFsShellReturnCode}} when running on Windows
if the logged-in user is a member of a group like Remote Desktop Users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable
[
https://issues.apache.org/jira/browse/HADOOP-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9768:
--
Attachment: HADOOP-9768-trunk.1.patch
I'm attaching a patch to allow the space character.
I've also added tests that specifically cover user and group names with spaces.
This way, we'll be able to catch regressions on any machine running the tests,
not just machines where the logged-in user happens to have spaces in the group
name.
chown and chgrp reject users and groups with spaces on platforms where spaces
are otherwise acceptable
--
Key: HADOOP-9768
URL: https://issues.apache.org/jira/browse/HADOOP-9768
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-9768-branch-1.1.patch, HADOOP-9768-trunk.1.patch
The chown and chgrp commands enforce a check on a valid set of characters for
user and group. The set of valid characters does not include space. On some
platforms (notably Windows), the space character is acceptable. We've seen
this cause test failures in {{TestFsShellReturnCode}} when running on Windows
if the logged-in user is a member of a group like Remote Desktop Users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9761) ViewFileSystem#rename fails when using DistributedFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-9761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718991#comment-13718991 ] Andrew Wang commented on HADOOP-9761: - They're checked in DistributedFileSystem#getPathName by calling FileSystem#checkPath (which does the validation you posted). All of our paths in DFS go through getPathName (or at least are supposed to). ViewFileSystem#rename fails when using DistributedFileSystem Key: HADOOP-9761 URL: https://issues.apache.org/jira/browse/HADOOP-9761 Project: Hadoop Common Issue Type: Bug Components: viewfs Affects Versions: 3.0.0, 2.1.0-beta Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hadoop-9761-1.patch ViewFileSystem currently passes unqualified paths (no scheme or authority) to underlying FileSystems when doing a rename. DistributedFileSystem symlink support added in HADOOP-9418 needs to qualify and check rename sources and destinations since cross-filesystem renames aren't supported, so this breaks in the following way - Default FS URI is configured to viewfs://viewfs - When doing a rename, ViewFileSystem checks to make sure both src and dst FileSystems are the same (which they are, both in same DFS), and then calls DistributedFileSystem#rename with unqualified remainder paths - Since these paths are unqualified, DFS qualifies them with the default FS to check that it can do the rename. This turns it into viewfs://viewfs/path - Since viewfs://viewfs is not the DFS's URI, DFS errors out the rename. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9756) Additional cleanup RPC code
[ https://issues.apache.org/jira/browse/HADOOP-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719005#comment-13719005 ] Tsz Wo (Nicholas), SZE commented on HADOOP-9756: Hi Junping, thanks for working on this. Please remove all other RPC.getServer(..) methods. They were deprecated by HADOOP-8754. You may replace the existing usage of them by RPC.Builder. For narrowing throws Exception, I notices that the changes are only happened in the tests. You may skip the them next time. I think it is better to focus on the main code and don't waste our energy on the style of the test code. :) Additional cleanup RPC code --- Key: HADOOP-9756 URL: https://issues.apache.org/jira/browse/HADOOP-9756 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Junping Du Assignee: Junping Du Priority: Minor Attachments: HADOOP-9756.patch HADOOP-9754 already did good job to address most of work for cleanup RPC code. Here is some additional work, include: - Remove some unused deprecated code. - Narrow throws Exception to throw some specific exception and remove some unnecessary exceptions - Fix a generic warning and correct spell issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-9652:
Attachment: hadoop-9652-3.patch
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
hadoop-9652-3.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719033#comment-13719033
]
Andrew Wang commented on HADOOP-9652:
-
Thanks for the review Colin, new patch attached.
bq. Newly created files get set to the user's primary group on UNIX.
TY, added that check back in.
bq. impossible to view relative symlink paths once they are created
I think FileSystem#getLinkTarget serves this purpose; targets are returned as
set.
bq. Shouldn't this be OS_TYPE_LINUX?
Truth, changed it to LINUX. It's not actually a positive Linux detection
though, since it's the default OSType.
bq. We should also really also file a follow-up bug to remove this and move to
a more portable solution when we get the green light to drop JDK6 support.
Definitely agree on this, there are lots of places we shell out to workaround
Java 6. I'll file a JIRA.
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
hadoop-9652-3.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable
[
https://issues.apache.org/jira/browse/HADOOP-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719035#comment-13719035
]
Hadoop QA commented on HADOOP-9768:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594052/HADOOP-9768-trunk.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2839//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2839//console
This message is automatically generated.
chown and chgrp reject users and groups with spaces on platforms where spaces
are otherwise acceptable
--
Key: HADOOP-9768
URL: https://issues.apache.org/jira/browse/HADOOP-9768
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-9768-branch-1.1.patch, HADOOP-9768-trunk.1.patch
The chown and chgrp commands enforce a check on a valid set of characters for
user and group. The set of valid characters does not include space. On some
platforms (notably Windows), the space character is acceptable. We've seen
this cause test failures in {{TestFsShellReturnCode}} when running on Windows
if the logged-in user is a member of a group like Remote Desktop Users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (HADOOP-9769) Remove org.apache.hadoop.fs.Stat when JDK6 support is dropped
[
https://issues.apache.org/jira/browse/HADOOP-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang moved HDFS-5030 to HADOOP-9769:
---
Affects Version/s: (was: 3.0.0)
3.0.0
Key: HADOOP-9769 (was: HDFS-5030)
Project: Hadoop Common (was: Hadoop HDFS)
Remove org.apache.hadoop.fs.Stat when JDK6 support is dropped
-
Key: HADOOP-9769
URL: https://issues.apache.org/jira/browse/HADOOP-9769
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Andrew Wang
Priority: Minor
HADOOP-9652 introduces a new class which shells out to stat(1) because of the
lack of lstat(2) in Java 6. Java 7 has support for reading symlink targets
via {{Files#readSymbolicLink}}.
When Hadoop drops Java 6 support, let's use this more portable method instead.
See:
http://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#readSymbolicLink(java.nio.file.Path)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9507:
--
Attachment: HADOOP-9507-trunk.2.patch
I'm uploading version 2 of the trunk patch to fix the javac deprecation
warning. Here is the incremental diff:
{code}
diff --git
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
hadoop-common-projec
index 2ee6c70..6b45fe3 100644
---
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
+++
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
@@ -332,7 +332,7 @@ public boolean rename(Path src, Path dst) throws
IOException {
// fails. Copy source content to destination and delete source.
if (this.exists(dst)) {
FileStatus sdst = this.getFileStatus(dst);
- if (sdst.isDir() dstFile.list().length == 0) {
+ if (sdst.isDirectory() dstFile.list().length == 0) {
if (LOG.isDebugEnabled()) {
LOG.debug(Copying contents of + src + to + dst);
}
{code}
On branch-1, we don't have {{FileStatus#isDirectory}}, and {{FileStatus#isDir}}
is not deprecated. Therefore, we don't need to update the branch-1 and
branch-1-win patches.
LocalFileSystem rename() is broken in some cases when destination exists
Key: HADOOP-9507
URL: https://issues.apache.org/jira/browse/HADOOP-9507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Mostafa Elhemali
Assignee: Chris Nauroth
Priority: Minor
Attachments: HADOOP-9507-branch-1.1.patch,
HADOOP-9507-branch-1-win.1.patch, HADOOP-9507.branch-1-win.patch,
HADOOP-9507-trunk.1.patch, HADOOP-9507-trunk.2.patch
The rename() method in RawLocalFileSystem uses FileUtil.copy() without
realizing that FileUtil.copy() has a special behavior that if you're copying
/foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
instead of overwriting it, which is not what rename() wants. So you end up
with weird behaviors like in this repro:
{code}
c:
cd \
md Foo
md Bar
md Foo\X
md Bar\X
hadoop fs -mv file:///c:/Foo file:///c:/Bar
{code}
At the end of this, you would expect to find only Bar\X, but you instead find
Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable
[
https://issues.apache.org/jira/browse/HADOOP-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9768:
--
Status: Open (was: Patch Available)
I'm canceling the patch for a moment. I realized that the allowable characters
are mentioned in the command usage output, so I'll want to update that too.
chown and chgrp reject users and groups with spaces on platforms where spaces
are otherwise acceptable
--
Key: HADOOP-9768
URL: https://issues.apache.org/jira/browse/HADOOP-9768
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-9768-branch-1.1.patch, HADOOP-9768-trunk.1.patch
The chown and chgrp commands enforce a check on a valid set of characters for
user and group. The set of valid characters does not include space. On some
platforms (notably Windows), the space character is acceptable. We've seen
this cause test failures in {{TestFsShellReturnCode}} when running on Windows
if the logged-in user is a member of a group like Remote Desktop Users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9652) RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
[
https://issues.apache.org/jira/browse/HADOOP-9652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719097#comment-13719097
]
Hadoop QA commented on HADOOP-9652:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594059/hadoop-9652-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2840//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2840//console
This message is automatically generated.
RawLocalFs#getFileLinkStatus does not fill in the link owner and mode
-
Key: HADOOP-9652
URL: https://issues.apache.org/jira/browse/HADOOP-9652
Project: Hadoop Common
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hadoop-9452-1.patch, hadoop-9652-2.patch,
hadoop-9652-3.patch
{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of
the symlink, but instead uses the owner and mode of the symlink target. If
the target can't be found, it fills in bogus values (the empty string and
FsPermission.getDefault) for these.
Symlinks have an owner distinct from the owner of the target they point to,
and getFileLinkStatus ought to expose this.
In some operating systems, symlinks can have a permission other than 0777.
We ought to expose this in RawLocalFilesystem and other places, although we
don't necessarily have to support this behavior in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9698) RPCv9 client must honor server's SASL negotiate response
[ https://issues.apache.org/jira/browse/HADOOP-9698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719107#comment-13719107 ] Sanjay Radia commented on HADOOP-9698: -- Q. wrt to this or the previous jira do we support the option of the client doing an INITIATE without the NEGOTIATE? RPCv9 client must honor server's SASL negotiate response Key: HADOOP-9698 URL: https://issues.apache.org/jira/browse/HADOOP-9698 Project: Hadoop Common Issue Type: Sub-task Components: ipc Affects Versions: 3.0.0, 2.1.0-beta Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-9698.patch As of HADOOP-9421, a RPCv9 server will advertise its authentication methods. This is meant to support features such as IP failover, better token selection, and interoperability in a heterogenous security environment. Currently the client ignores the negotiate response and just blindly attempts to authenticate instead of choosing a mutually agreeable auth method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9507) LocalFileSystem rename() is broken in some cases when destination exists
[
https://issues.apache.org/jira/browse/HADOOP-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719116#comment-13719116
]
Hadoop QA commented on HADOOP-9507:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594069/HADOOP-9507-trunk.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2841//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2841//console
This message is automatically generated.
LocalFileSystem rename() is broken in some cases when destination exists
Key: HADOOP-9507
URL: https://issues.apache.org/jira/browse/HADOOP-9507
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Mostafa Elhemali
Assignee: Chris Nauroth
Priority: Minor
Attachments: HADOOP-9507-branch-1.1.patch,
HADOOP-9507-branch-1-win.1.patch, HADOOP-9507.branch-1-win.patch,
HADOOP-9507-trunk.1.patch, HADOOP-9507-trunk.2.patch
The rename() method in RawLocalFileSystem uses FileUtil.copy() without
realizing that FileUtil.copy() has a special behavior that if you're copying
/foo to /bar and /bar exists and is a directory, it'll copy /foo inside /bar
instead of overwriting it, which is not what rename() wants. So you end up
with weird behaviors like in this repro:
{code}
c:
cd \
md Foo
md Bar
md Foo\X
md Bar\X
hadoop fs -mv file:///c:/Foo file:///c:/Bar
{code}
At the end of this, you would expect to find only Bar\X, but you instead find
Bar\X\X.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9770) RetryCache#state need not be volatile
[ https://issues.apache.org/jira/browse/HADOOP-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9770: Assignee: Suresh Srinivas RetryCache#state need not be volatile - Key: HADOOP-9770 URL: https://issues.apache.org/jira/browse/HADOOP-9770 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 2.1.0-beta Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Minor See the comment - https://issues.apache.org/jira/browse/HDFS-4979?focusedCommentId=13719111page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13719111 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9770) RetryCache#state need not be volatile
Suresh Srinivas created HADOOP-9770: --- Summary: RetryCache#state need not be volatile Key: HADOOP-9770 URL: https://issues.apache.org/jira/browse/HADOOP-9770 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 2.1.0-beta Reporter: Suresh Srinivas Priority: Minor See the comment - https://issues.apache.org/jira/browse/HDFS-4979?focusedCommentId=13719111page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13719111 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9770) RetryCache#state need not be volatile
[ https://issues.apache.org/jira/browse/HADOOP-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9770: Status: Patch Available (was: Open) RetryCache#state need not be volatile - Key: HADOOP-9770 URL: https://issues.apache.org/jira/browse/HADOOP-9770 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 2.1.0-beta Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Minor Attachments: HADOOP-9770.patch See the comment - https://issues.apache.org/jira/browse/HDFS-4979?focusedCommentId=13719111page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13719111 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9770) RetryCache#state need not be volatile
[ https://issues.apache.org/jira/browse/HADOOP-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9770: Attachment: HADOOP-9770.patch RetryCache#state need not be volatile - Key: HADOOP-9770 URL: https://issues.apache.org/jira/browse/HADOOP-9770 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 2.1.0-beta Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Minor Attachments: HADOOP-9770.patch See the comment - https://issues.apache.org/jira/browse/HDFS-4979?focusedCommentId=13719111page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13719111 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9770) RetryCache#state need not be volatile
[
https://issues.apache.org/jira/browse/HADOOP-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719158#comment-13719158
]
Hadoop QA commented on HADOOP-9770:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594081/HADOOP-9770.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2842//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2842//console
This message is automatically generated.
RetryCache#state need not be volatile
-
Key: HADOOP-9770
URL: https://issues.apache.org/jira/browse/HADOOP-9770
Project: Hadoop Common
Issue Type: Improvement
Components: util
Affects Versions: 2.1.0-beta
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
Priority: Minor
Attachments: HADOOP-9770.patch
See the comment -
https://issues.apache.org/jira/browse/HDFS-4979?focusedCommentId=13719111page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13719111
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8990) Clean up RPC protocol for consistency
[ https://issues.apache.org/jira/browse/HADOOP-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719200#comment-13719200 ] Binglin Chang commented on HADOOP-8990: --- bq. The current fields are required in our rpc protobufs - shall we make all or some of them optional? Agree, it is good for compatibility Clean up RPC protocol for consistency -- Key: HADOOP-8990 URL: https://issues.apache.org/jira/browse/HADOOP-8990 Project: Hadoop Common Issue Type: Improvement Reporter: Binglin Chang Assignee: Sanjay Radia There are a few inconsistencies in the current RPC protocol that make new client implementation (especially non-blocking) unnecessarily hard. For example: # proto file naming RpcPayloadHeader.proto include not only RpcPayLoadHeaderProto, but also RpcResponseHeaderProto, which is irrelevant to the file name. hadoop_rpc.proto only include HadoopRpcRequestProto, and the filename hadoop_rpc is strange comparing to other .proto file names. How about merge those two file into HadoopRpc.proto? # proto class naming In rpc request RpcPayloadHeaderProto includes callId, but in rpc response callId is included in RpcResponseHeaderProto, and there is also HadoopRpcRequestProto, this is just too confusing. # The rpc system is not fully protobuf based, there are still some Writables: RpcRequestWritable, RpcResponseWritable, rpc response exception name, stack trace string and sasl request/response. RpcRequestWritable uses protobuf style varint32 prefix, but RpcResponseWritable uses int32 prefix, why this inconsistency? Currently rpc request is splitted into length, PayLoadHeader and PayLoad, and response into RpcResponseHeader, response and error message. Wrapping request and response into single RequestProto and ResponseProto would be better, since this gives a formal complete wire format definition. These issues above make it very confusing and hard for developers to use these rpc interfaces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9609) Remove sh dependency of bin-package target
[ https://issues.apache.org/jira/browse/HADOOP-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9609: -- Component/s: build Target Version/s: 1-win Hadoop Flags: Reviewed +1 for the patch. I believe the latest patch has addressed the last round of feedback. I'm going to wait a day before committing, just in case [~ivanmi] has additional feedback. Remove sh dependency of bin-package target -- Key: HADOOP-9609 URL: https://issues.apache.org/jira/browse/HADOOP-9609 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1-win Reporter: Chuan Liu Assignee: Chuan Liu Priority: Minor Attachments: HADOOP-9609-branch-1-win-2.patch, HADOOP-9609-branch-1-win-3.patch, HADOOP-9609-branch-1-win.patch In Ant package target, we no longer use packageBinNativeHadoop.sh to place native library binaries. However, the same script is still present in the bin-package target. We should remove bin-package target's dependency on sh to keep it consistent with package target. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable
[
https://issues.apache.org/jira/browse/HADOOP-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9768:
--
Attachment: HADOOP-9768-trunk.2.patch
I'm uploading new patches that also update the list of valid characters
displayed by command usage. While I was in there, I corrected a typo too.
chown and chgrp reject users and groups with spaces on platforms where spaces
are otherwise acceptable
--
Key: HADOOP-9768
URL: https://issues.apache.org/jira/browse/HADOOP-9768
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-9768-branch-1.1.patch,
HADOOP-9768-branch-1.2.patch, HADOOP-9768-trunk.1.patch,
HADOOP-9768-trunk.2.patch
The chown and chgrp commands enforce a check on a valid set of characters for
user and group. The set of valid characters does not include space. On some
platforms (notably Windows), the space character is acceptable. We've seen
this cause test failures in {{TestFsShellReturnCode}} when running on Windows
if the logged-in user is a member of a group like Remote Desktop Users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable
[
https://issues.apache.org/jira/browse/HADOOP-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9768:
--
Attachment: HADOOP-9768-branch-1.2.patch
chown and chgrp reject users and groups with spaces on platforms where spaces
are otherwise acceptable
--
Key: HADOOP-9768
URL: https://issues.apache.org/jira/browse/HADOOP-9768
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-9768-branch-1.1.patch,
HADOOP-9768-branch-1.2.patch, HADOOP-9768-trunk.1.patch,
HADOOP-9768-trunk.2.patch
The chown and chgrp commands enforce a check on a valid set of characters for
user and group. The set of valid characters does not include space. On some
platforms (notably Windows), the space character is acceptable. We've seen
this cause test failures in {{TestFsShellReturnCode}} when running on Windows
if the logged-in user is a member of a group like Remote Desktop Users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable
[
https://issues.apache.org/jira/browse/HADOOP-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9768:
--
Status: Patch Available (was: Open)
chown and chgrp reject users and groups with spaces on platforms where spaces
are otherwise acceptable
--
Key: HADOOP-9768
URL: https://issues.apache.org/jira/browse/HADOOP-9768
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-9768-branch-1.1.patch,
HADOOP-9768-branch-1.2.patch, HADOOP-9768-trunk.1.patch,
HADOOP-9768-trunk.2.patch
The chown and chgrp commands enforce a check on a valid set of characters for
user and group. The set of valid characters does not include space. On some
platforms (notably Windows), the space character is acceptable. We've seen
this cause test failures in {{TestFsShellReturnCode}} when running on Windows
if the logged-in user is a member of a group like Remote Desktop Users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable
[
https://issues.apache.org/jira/browse/HADOOP-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719255#comment-13719255
]
Chuan Liu commented on HADOOP-9768:
---
+1
Look good to me! Thanks for fix the problem, Chris!
chown and chgrp reject users and groups with spaces on platforms where spaces
are otherwise acceptable
--
Key: HADOOP-9768
URL: https://issues.apache.org/jira/browse/HADOOP-9768
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-9768-branch-1.1.patch,
HADOOP-9768-branch-1.2.patch, HADOOP-9768-trunk.1.patch,
HADOOP-9768-trunk.2.patch
The chown and chgrp commands enforce a check on a valid set of characters for
user and group. The set of valid characters does not include space. On some
platforms (notably Windows), the space character is acceptable. We've seen
this cause test failures in {{TestFsShellReturnCode}} when running on Windows
if the logged-in user is a member of a group like Remote Desktop Users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9768) chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable
[
https://issues.apache.org/jira/browse/HADOOP-9768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719258#comment-13719258
]
Arpit Agarwal commented on HADOOP-9768:
---
+1 for the trunk patch. Verified on Windows and OS X. Nice use of
{{assertIllegalArguments}}.
chown and chgrp reject users and groups with spaces on platforms where spaces
are otherwise acceptable
--
Key: HADOOP-9768
URL: https://issues.apache.org/jira/browse/HADOOP-9768
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0, 1-win, 2.1.0-beta, 1.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-9768-branch-1.1.patch,
HADOOP-9768-branch-1.2.patch, HADOOP-9768-trunk.1.patch,
HADOOP-9768-trunk.2.patch
The chown and chgrp commands enforce a check on a valid set of characters for
user and group. The set of valid characters does not include space. On some
platforms (notably Windows), the space character is acceptable. We've seen
this cause test failures in {{TestFsShellReturnCode}} when running on Windows
if the logged-in user is a member of a group like Remote Desktop Users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
