[jira] [Commented] (HADOOP-9064) Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction
[
https://issues.apache.org/jira/browse/HADOOP-9064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505361#comment-13505361
]
Hudson commented on HADOOP-9064:
Integrated in Hadoop-Yarn-trunk #50 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/50/])
HADOOP-9064. Ameding CHANGES.txt with correct JIRA ID, wrongfully used
HADOOP-9084 before (tucu) (Revision 1414347)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414347
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
Augment DelegationTokenRenewer API to cancel the tokens on calls to
removeRenewAction
-
Key: HADOOP-9064
URL: https://issues.apache.org/jira/browse/HADOOP-9064
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Fix For: 2.0.3-alpha
Attachments: hadoop-9064.patch, hadoop-9064.patch, hadoop-9064.patch,
hadoop-9064.patch
Post HADOOP-9049, FileSystems register with DelegationTokenRenewer (a
singleton), to renew tokens.
To avoid a bunch of defunct tokens clog the NN, we should augment the API to
{{#removeRenewAction(boolean cancel)}} and cancel the token appropriately.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9084) TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
[ https://issues.apache.org/jira/browse/HADOOP-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505360#comment-13505360 ] Hudson commented on HADOOP-9084: Integrated in Hadoop-Yarn-trunk #50 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/50/]) HADOOP-9064. Ameding CHANGES.txt with correct JIRA ID, wrongfully used HADOOP-9084 before (tucu) (Revision 1414347) HADOOP-9084. Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction. (kkambatl via tucu) (Revision 1414326) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414347 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414326 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDelegationTokenRenewer.java TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem) Key: HADOOP-9084 URL: https://issues.apache.org/jira/browse/HADOOP-9084 Project: Hadoop Common Issue Type: Bug Components: filecache, fs, native Affects Versions: 1.0.3, 1.0.4 Reporter: giovanni delussu Assignee: giovanni delussu Priority: Critical Fix For: 1.0.3, 1.0.4 Attachments: PATCH-HADOOP-9084-origin-branch-1.0.patch When running a job who uses TotalOrderPartitioner (like TeraSort or BulkImport of HBase) on hadoop running on top of gpfs (instead of hdfs) the program fails to find the file _partition.lst because is looking for it in the wrong directory. The confusion is between local fs meaning not hdfs and local meaning distributed fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7868) Hadoop native fails to compile when default linker option is -Wl,--as-needed
[ https://issues.apache.org/jira/browse/HADOOP-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505422#comment-13505422 ] Hudson commented on HADOOP-7868: Integrated in Hadoop-Hdfs-0.23-Build #449 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/449/]) HADOOP-7868. Hadoop native fails to compile when default linker option is -Wl,--as-needed. (Trevor Robinson via tgraves) (Revision 1414370) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414370 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/acinclude.m4 * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/native/configure.ac Hadoop native fails to compile when default linker option is -Wl,--as-needed Key: HADOOP-7868 URL: https://issues.apache.org/jira/browse/HADOOP-7868 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 1.0.0, 2.0.0-alpha Environment: Ubuntu Precise, Ubuntu Oneiric, Debian Unstable Reporter: James Page Assignee: Trevor Robinson Fix For: 1.2.0, 2.0.2-alpha, 0.23.6 Attachments: hadoop-7868-b1.txt, HADOOP-7868.patch, HADOOP-7868-portable.patch Recent releases of Ubuntu and Debian have switched to using --as-needed as default when linking binaries. As a result the AC_COMPUTE_NEEDED_DSO fails to find the required DSO names during execution of configure resulting in a build failure. Explicitly using -Wl,--no-as-needed in this macro when required resolves this issue. See http://wiki.debian.org/ToolChain/DSOLinking for a few more details -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505509#comment-13505509 ] Daryn Sharp commented on HADOOP-8989: - Sorry, this fell off my radar. I've been swamped. bq. Has anyone studied the impact on the NN yet? W/o having reviewed the patch, I initially don't think this could be any worse than ls -R. hadoop dfs -find feature Key: HADOOP-8989 URL: https://issues.apache.org/jira/browse/HADOOP-8989 Project: Hadoop Common Issue Type: New Feature Reporter: Marco Nicosia Assignee: Jonathan Allen Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch Both sysadmins and users make frequent use of the unix 'find' command, but Hadoop has no correlate. Without this, users are writing scripts which make heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs -lsr is somewhat taxing on the NameNode, and a really slow experience on the client side. Possibly an in-NameNode find operation would be only a bit more taxing on the NameNode, but significantly faster from the client's point of view? The minimum set of options I can think of which would make a Hadoop find command generally useful is (in priority order): * -type (file or directory, for now) * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) * -print0 (for piping to xargs -0) * -depth * -owner/-group (and -nouser/-nogroup) * -name (allowing for shell pattern, or even regex?) * -perm * -size One possible special case, but could possibly be really cool if it ran from within the NameNode: * -delete The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow. Lower priority, some people do use operators, mostly to execute -or searches such as: * find / \(-nouser -or -nogroup\) Finally, I thought I'd include a link to the [Posix spec for find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9046: --- Attachment: HADOOP-9046--c.patch In the new version of patch HADOOP-9046--c.patch the following is done: 1) fixed the sync problem found by Robert: now the method org.apache.hadoop.fs.DelegationTokenRenewer#removeRenewAction(T) will not block for long time. 2) in the test all the time periods made scalable relative one constant. 3) the test made fater: now it takes ~7 seconds. provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT -- Key: HADOOP-9046 URL: https://issues.apache.org/jira/browse/HADOOP-9046 Project: Hadoop Common Issue Type: Test Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23.patch, HADOOP-9046--c.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero coverage in entire cumulative test run. Provide test(s) to cover this class. Note: the request submitted to HDFS project because the class likely to be tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
[
https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505561#comment-13505561
]
Hadoop QA commented on HADOOP-9046:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555168/HADOOP-9046--c.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1834//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1834//console
This message is automatically generated.
provide unit-test coverage of class
org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
--
Key: HADOOP-9046
URL: https://issues.apache.org/jira/browse/HADOOP-9046
Project: Hadoop Common
Issue Type: Test
Reporter: Ivan A. Veselovsky
Assignee: Ivan A. Veselovsky
Priority: Minor
Attachments: HADOOP-9046-branch-0.23-over-9049.patch,
HADOOP-9046-branch-0.23.patch, HADOOP-9046--c.patch,
HADOOP-9046-over-9049.patch, HADOOP-9046.patch
The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero
coverage in entire cumulative test run. Provide test(s) to cover this class.
Note: the request submitted to HDFS project because the class likely to be
tested by tests in that project.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9044) add FindClass main class to provide classpath checking of installations
[ https://issues.apache.org/jira/browse/HADOOP-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-9044: --- Attachment: HADOOP-9044.patch patch against trunk add FindClass main class to provide classpath checking of installations --- Key: HADOOP-9044 URL: https://issues.apache.org/jira/browse/HADOOP-9044 Project: Hadoop Common Issue Type: New Feature Components: util Affects Versions: 1.1.0, 2.0.3-alpha Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Attachments: HADOOP-9044.patch, HADOOP-9044.patch It's useful in postflight checking of a hadoop installation to verify that classes load, especially codes with external JARs and native codecs. An entry point designed to load a named class and create an instance of that class can do this -and be invoked from any shell script or tool that does the installation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9044) add FindClass main class to provide classpath checking of installations
[ https://issues.apache.org/jira/browse/HADOOP-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-9044: --- Affects Version/s: 2.0.3-alpha Status: Patch Available (was: In Progress) add FindClass main class to provide classpath checking of installations --- Key: HADOOP-9044 URL: https://issues.apache.org/jira/browse/HADOOP-9044 Project: Hadoop Common Issue Type: New Feature Components: util Affects Versions: 1.1.0, 2.0.3-alpha Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Attachments: HADOOP-9044.patch, HADOOP-9044.patch It's useful in postflight checking of a hadoop installation to verify that classes load, especially codes with external JARs and native codecs. An entry point designed to load a named class and create an instance of that class can do this -and be invoked from any shell script or tool that does the installation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9046: --- Attachment: HADOOP-9046-branch-0.23--c.patch Patch HADOOP-9046-branch-0.23--c.patch ports the changes to branch-0.23. The trunk patch HADOOP-9046--c.patch is applicable to branch-2 as well. provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT -- Key: HADOOP-9046 URL: https://issues.apache.org/jira/browse/HADOOP-9046 Project: Hadoop Common Issue Type: Test Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9046-branch-0.23--c.patch, HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23.patch, HADOOP-9046--c.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero coverage in entire cumulative test run. Provide test(s) to cover this class. Note: the request submitted to HDFS project because the class likely to be tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9046: --- Affects Version/s: 0.23.6 2.0.3-alpha 3.0.0 provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT -- Key: HADOOP-9046 URL: https://issues.apache.org/jira/browse/HADOOP-9046 Project: Hadoop Common Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9046-branch-0.23--c.patch, HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23.patch, HADOOP-9046--c.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero coverage in entire cumulative test run. Provide test(s) to cover this class. Note: the request submitted to HDFS project because the class likely to be tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
[
https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505661#comment-13505661
]
Robert Joseph Evans commented on HADOOP-9046:
-
It is looking good, and I like how much faster the tests run.
I have a few more comments.
# {code}TODO: should we reject the addition if an action for this 'fs' is
already present in the queue?{code} If this is an issue then we should file a
JIRA to fix it, otherwise please remove the TODO. I am leaning more towards
filing a separate JIRA.
# I don't really like the name lock0 and available0. I would rather have them
be named something more descriptive. This is very minor
# in removeRenewAction we are no longer canceling the token when it is removed.
I am not sure if that is intentional or not, but it definitely changes the
functionality of the method.
# I am not sure that renewerThreadStarted.compairAndSet is that much better
then isAlive(). isAlive has the problem that if the thread stopped, calling
add again would result in an IllegalThreadStateException, but compairAndSet has
the problem that if a new thread cannot be created (Which I have seen happen)
then only the first call to add will get an exception all other calls to add
will look like they succeeded but no renewal will ever happen. I would rather
see errors happen on other calls to add then to have them silently fail.
provide unit-test coverage of class
org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
--
Key: HADOOP-9046
URL: https://issues.apache.org/jira/browse/HADOOP-9046
Project: Hadoop Common
Issue Type: Test
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
Reporter: Ivan A. Veselovsky
Assignee: Ivan A. Veselovsky
Priority: Minor
Attachments: HADOOP-9046-branch-0.23--c.patch,
HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23.patch,
HADOOP-9046--c.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch
The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero
coverage in entire cumulative test run. Provide test(s) to cover this class.
Note: the request submitted to HDFS project because the class likely to be
tested by tests in that project.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9101) make s3n NativeFileSystemStore interface public instead of package-private
Steve Loughran created HADOOP-9101:
--
Summary: make s3n NativeFileSystemStore interface public instead
of package-private
Key: HADOOP-9101
URL: https://issues.apache.org/jira/browse/HADOOP-9101
Project: Hadoop Common
Issue Type: Improvement
Components: fs/s3
Affects Versions: 3.0.0
Reporter: Steve Loughran
Priority: Trivial
It would be easier to implement new blockstore filesystems if the the
{{NativeFileSystemStore} and dependent classes in the
{{org.apache.hadoop.fs.s3native}} package were public -currently you need to
put them into the s3 directory.
They could be made public with the appropriate scope attribute. Internal?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9078) enhance unit-test coverage of class org.apache.hadoop.fs.FileContext
[ https://issues.apache.org/jira/browse/HADOOP-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9078: --- Attachment: HADOOP-9078-branch-2.patch HADOOP-9078-branch-0.23.patch enhance unit-test coverage of class org.apache.hadoop.fs.FileContext Key: HADOOP-9078 URL: https://issues.apache.org/jira/browse/HADOOP-9078 Project: Hadoop Common Issue Type: Test Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Attachments: HADOOP-9078-branch-0.23.patch, HADOOP-9078-branch-2.patch, HADOOP-9078.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9078) enhance unit-test coverage of class org.apache.hadoop.fs.FileContext
[ https://issues.apache.org/jira/browse/HADOOP-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9078: --- Attachment: HADOOP-9078.patch HADOOP-9078 is for trunk, other patches are for the branches respectively. enhance unit-test coverage of class org.apache.hadoop.fs.FileContext Key: HADOOP-9078 URL: https://issues.apache.org/jira/browse/HADOOP-9078 Project: Hadoop Common Issue Type: Test Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Attachments: HADOOP-9078-branch-0.23.patch, HADOOP-9078-branch-2.patch, HADOOP-9078.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9078) enhance unit-test coverage of class org.apache.hadoop.fs.FileContext
[ https://issues.apache.org/jira/browse/HADOOP-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9078: --- Affects Version/s: 0.23.6 2.0.3-alpha 3.0.0 Status: Patch Available (was: Open) enhance unit-test coverage of class org.apache.hadoop.fs.FileContext Key: HADOOP-9078 URL: https://issues.apache.org/jira/browse/HADOOP-9078 Project: Hadoop Common Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Attachments: HADOOP-9078-branch-0.23.patch, HADOOP-9078-branch-2.patch, HADOOP-9078.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated HADOOP-8981: -- Attachment: HADOOP-8981-branch-trunk-win.1.patch Attached the patch that addresses the test fails on windows. Add a test case to verify the callback correctly, abd test case to verify the stop call immediately TestMetricsSystemImpl fails on Windows -- Key: HADOOP-8981 URL: https://issues.apache.org/jira/browse/HADOOP-8981 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Arpit Agarwal Attachments: HADOOP-8981-branch-trunk-win.1.patch The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9100) Fix TODO items in FsShell
[ https://issues.apache.org/jira/browse/HADOOP-9100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505846#comment-13505846 ] Eli Collins commented on HADOOP-9100: - See HADOOP-8690 for one of them. Fix TODO items in FsShell - Key: HADOOP-9100 URL: https://issues.apache.org/jira/browse/HADOOP-9100 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Minor Bunch of TODO items in FsShell needs to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9090) Refactor MetricsSystemImpl to allow for an on-demand publish system
[ https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Elhemali updated HADOOP-9090: - Attachment: HADOOP-9090.justEnhanceDefaultImpl.3.patch Thanks Luke. It's a fundamentally hard problem: if we assume that putMetrics() from arbitrary sinks can hang, then we'll have to interrupt the thread that doing the work and hope that unsticks it (that's how thread pools cancel work for example). Having said that, I got uneasy about using stop(), and I agree with your comment that I was creating too many threads willy-nilly. The new patch thus does two things: 1. Creates a single thread (if needed) for on-demand publishing per sink, and reuses that for subsequent publishing. This should cut down considerably on how many threads are created. 2. If it sees a sink timeout, it interrupts the worker thread but doesn't stop() it (joins it and hangs with it if needed). I personally think that's the most reliable thing we can do. One alternative as you say is to mark the worker thread as a daemon, so we don't have to join it and hang with it if it hangs, but just abandon it. The problem there is that sinks may be doing things to external systems that will leave them in inconsistent states if the process just shuts down while they're still working with no warning. Let me know what you think. Refactor MetricsSystemImpl to allow for an on-demand publish system --- Key: HADOOP-9090 URL: https://issues.apache.org/jira/browse/HADOOP-9090 Project: Hadoop Common Issue Type: New Feature Components: metrics Reporter: Mostafa Elhemali Priority: Minor Attachments: HADOOP-9090.2.patch, HADOOP-9090.justEnhanceDefaultImpl.2.patch, HADOOP-9090.justEnhanceDefaultImpl.3.patch, HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch We have a need to publish metrics out of some short-living processes, which is not really well-suited to the current metrics system implementation which periodically publishes metrics asynchronously (a behavior that works great for long-living processes). Of course I could write my own metrics system, but it seems like such a waste to rewrite all the awesome code currently in the MetricsSystemImpl and supporting classes. The way I'm proposing to solve this is to: 1. Refactor the MetricsSystemImpl class into an abstract base MetricsSystemImpl class (common configuration and other code) and a concrete PeriodicPublishMetricsSystemImpl class (timer thread). 2. Refactor the MetricsSinkAdapter class into an abstract base MetricsSinkAdapter class (common configuration and other code) and a concrete AsyncMetricsSinkAdapter class (asynchronous publishing using the SinkQueue). 3. Derive a new simple class OnDemandPublishMetricsSystemImpl from MetricsSystemImpl, that just exposes a synchronous publish() method to do all the work. 4. Derive a SyncMetricsSinkAdapter class from MetricsSinkAdapter to just synchronously push metrics to the underlying sink. Does that sound reasonable? I'll attach the patch with all this coded up and simple tests (could use some polish I guess, but wanted to get everyone's opinion first). Notice that this is somewhat of a breaking change since MetricsSystemImpl is public (although it's marked with InterfaceAudience.Private); if the breaking change is a problem I could just rename the refactored classes so that PeriodicPublishMetricsSystemImpl is still called MetricsSystemImpl (and MetricsSystemImpl - BaseMetricsSystemImpl). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506017#comment-13506017 ] Jonathan Allen commented on HADOOP-8989: Allen - I haven't done any specific studies of the NN impact but I would expect it to be the same as any of the existing recursive command (e.g. ls -R). The code uses the existing recursion code and applies the expressions to the returned FileStatus on the client side. The -exec option obviously impacts the NN but no more than if you ran the command directly (though you don't have the JVM start-up overhead so the rate of NN messages will be higher). Wolfgang - I'll look at adding those option expressions in (I got to the point where it looked like I'd be adding expressions forever so I decided to stop and see what people asked for). Daryn - if you do get a chance to review this then the main code is pretty much finished (though I need to sort out the formatting before submitting properly) but I've reworked the unit tests since uploading the last patch (I'll upload a new at the weekend). I still need to update the documentation web page and add some stuff in the CLI test suites. hadoop dfs -find feature Key: HADOOP-8989 URL: https://issues.apache.org/jira/browse/HADOOP-8989 Project: Hadoop Common Issue Type: New Feature Reporter: Marco Nicosia Assignee: Jonathan Allen Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch Both sysadmins and users make frequent use of the unix 'find' command, but Hadoop has no correlate. Without this, users are writing scripts which make heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs -lsr is somewhat taxing on the NameNode, and a really slow experience on the client side. Possibly an in-NameNode find operation would be only a bit more taxing on the NameNode, but significantly faster from the client's point of view? The minimum set of options I can think of which would make a Hadoop find command generally useful is (in priority order): * -type (file or directory, for now) * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) * -print0 (for piping to xargs -0) * -depth * -owner/-group (and -nouser/-nogroup) * -name (allowing for shell pattern, or even regex?) * -perm * -size One possible special case, but could possibly be really cool if it ran from within the NameNode: * -delete The hadoop dfs -lsr | hadoop dfs -rm cycle is really, really slow. Lower priority, some people do use operators, mostly to execute -or searches such as: * find / \(-nouser -or -nogroup\) Finally, I thought I'd include a link to the [Posix spec for find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506031#comment-13506031 ] Arpit Agarwal commented on HADOOP-8981: --- Tabs should be replaced with two spaces. Looks fine otherwise. TestMetricsSystemImpl fails on Windows -- Key: HADOOP-8981 URL: https://issues.apache.org/jira/browse/HADOOP-8981 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Arpit Agarwal Attachments: HADOOP-8981-branch-trunk-win.1.patch The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506040#comment-13506040 ] Ivan Mitic commented on HADOOP-8981: Is this the same issue as reported by HADOOP-9057? TestMetricsSystemImpl fails on Windows -- Key: HADOOP-8981 URL: https://issues.apache.org/jira/browse/HADOOP-8981 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Arpit Agarwal Attachments: HADOOP-8981-branch-trunk-win.1.patch The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9090) Refactor MetricsSystemImpl to allow for an on-demand publish system
[
https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506045#comment-13506045
]
Luke Lu commented on HADOOP-9090:
-
I don't think you can control what happens to external systems, which _should_
handle arbitrary connection errors etc by unexpected client (even
router/firewall) shutdown. Another problem of the new patch is that you're
creating a latch and a semaphore per record (vs per MetricsBuffer) and there
can be hundreds (up to a few thousands) of records per put. If the sink hangs,
you'll be recreating new thread/latch/semaphore per record and the user
perceived timeout would be configured timeout * number of records. Another
issue is that hanging can happen in sink.flush as well.
Why not do the simple notification in the existing code like the following
(untested sketch)?:
{code}
boolean oobPut;
// illustration only, should be in the ctor after retry* variables are defined
final long OOB_PUT_TIMEOUT = retryDelay * Math.pow(retryBackoff, retryCount) *
1000;
synchronized void putMetricsImmediate(MetricsBuffer mb) {
if (!oobPut) {
oobPut = true;
if (queue.enqueue(buffer)) {
wait(OOB_PUT_TIMEOUT);
oobPut = false;
} // otherwise queue is full due to sink issues anyway.
} else { // another oobPut in progress
wait(OOB_PUT_TIMEOUT);
oobPut = false; // just in case
}
}
// after queue.consumeAll(this); in publishMetricsFromQueue (needs to be
synchronized now)
if (oobPut) {
notifyAll();
}
{code}
Now you get all the retry/timeout logic for free :)
Refactor MetricsSystemImpl to allow for an on-demand publish system
---
Key: HADOOP-9090
URL: https://issues.apache.org/jira/browse/HADOOP-9090
Project: Hadoop Common
Issue Type: New Feature
Components: metrics
Reporter: Mostafa Elhemali
Priority: Minor
Attachments: HADOOP-9090.2.patch,
HADOOP-9090.justEnhanceDefaultImpl.2.patch,
HADOOP-9090.justEnhanceDefaultImpl.3.patch,
HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch
We have a need to publish metrics out of some short-living processes, which
is not really well-suited to the current metrics system implementation which
periodically publishes metrics asynchronously (a behavior that works great
for long-living processes). Of course I could write my own metrics system,
but it seems like such a waste to rewrite all the awesome code currently in
the MetricsSystemImpl and supporting classes.
The way I'm proposing to solve this is to:
1. Refactor the MetricsSystemImpl class into an abstract base
MetricsSystemImpl class (common configuration and other code) and a concrete
PeriodicPublishMetricsSystemImpl class (timer thread).
2. Refactor the MetricsSinkAdapter class into an abstract base
MetricsSinkAdapter class (common configuration and other code) and a concrete
AsyncMetricsSinkAdapter class (asynchronous publishing using the SinkQueue).
3. Derive a new simple class OnDemandPublishMetricsSystemImpl from
MetricsSystemImpl, that just exposes a synchronous publish() method to do all
the work.
4. Derive a SyncMetricsSinkAdapter class from MetricsSinkAdapter to just
synchronously push metrics to the underlying sink.
Does that sound reasonable? I'll attach the patch with all this coded up and
simple tests (could use some polish I guess, but wanted to get everyone's
opinion first). Notice that this is somewhat of a breaking change since
MetricsSystemImpl is public (although it's marked with
InterfaceAudience.Private); if the breaking change is a problem I could just
rename the refactored classes so that PeriodicPublishMetricsSystemImpl is
still called MetricsSystemImpl (and MetricsSystemImpl -
BaseMetricsSystemImpl).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506049#comment-13506049 ] Arpit Agarwal commented on HADOOP-8981: --- Yes, it is the same. Your fix is very similar to what Chuan has submitted i.e. adding a short sleep before the verification. TestMetricsSystemImpl fails on Windows -- Key: HADOOP-8981 URL: https://issues.apache.org/jira/browse/HADOOP-8981 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Arpit Agarwal Attachments: HADOOP-8981-branch-trunk-win.1.patch The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9102) winutils task isAlive does not return a non-zero exit code if the requested task is not alive
Chris Nauroth created HADOOP-9102: - Summary: winutils task isAlive does not return a non-zero exit code if the requested task is not alive Key: HADOOP-9102 URL: https://issues.apache.org/jira/browse/HADOOP-9102 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Work on YARN-233 noted that winutils task isAlive does not return a non-zero exit code if the requested task is not alive. This is inconsistent with the equivalent check on Unix, kill -0, which uses a non-zero exit code to communicate status. By changing winutils task isAlive to return a non-zero exit code, we can make the error handling code on the Java side consistent, avoiding the need for additional if (WINDOWS) checks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9057) TestMetricsSystemImpl.testInitFirst fails intermittently
[ https://issues.apache.org/jira/browse/HADOOP-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506103#comment-13506103 ] Luke Lu commented on HADOOP-9057: - Mostafa: the timeout in mockito verify is a good idea. If you can provide trunk patch and branch-1 patches, I'll commit them. TestMetricsSystemImpl.testInitFirst fails intermittently Key: HADOOP-9057 URL: https://issues.apache.org/jira/browse/HADOOP-9057 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-9057.branch-1-win.metricsrace.patch Error Message Wanted but not invoked: metricsSink.putMetrics(Capturing argument); - at org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) Actually, there were zero interactions with this mock. Stacktrace Wanted but not invoked: metricsSink.putMetrics(Capturing argument); - at org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) Actually, there were zero interactions with this mock. at org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.run(JUnit45AndHigherRunnerImpl.java:37) at org.mockito.runners.MockitoJUnitRunner.run(MockitoJUnitRunner.java:62) Standard Output 2012-10-04 11:43:55,641 INFO impl.MetricsConfig (MetricsConfig.java:loadFirst(99)) - loaded properties from hadoop-metrics2-test.properties -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9090) Refactor MetricsSystemImpl to allow for an on-demand publish system
[ https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Elhemali updated HADOOP-9090: - Attachment: HADOOP-9090.justEnhanceDefaultImpl.4.patch (I highly appreciate the responsive feedback I'm getting from you Luke.) Thanks - you raise good points. My fear was that do we want the immediate sinks to be queued up or not, and do we want to leave the queue thread hanging if the sink times out or not. I think though I was swayed to your way of thinking by two things: a) the flush() problem you point out and b) that my previous patch would've left sinks possible called by multiple threads instead of just the queue thread which can be bad. I've implemented the spirit of what you said. I opted though to have the putMetricsImmediate() wait for its specific MetricsBuffer since I figured the way you had it may end up in race conditions if multiple threads are calling publishMetricsNow() at the same time. Let me know what you think - thanks! Refactor MetricsSystemImpl to allow for an on-demand publish system --- Key: HADOOP-9090 URL: https://issues.apache.org/jira/browse/HADOOP-9090 Project: Hadoop Common Issue Type: New Feature Components: metrics Reporter: Mostafa Elhemali Priority: Minor Attachments: HADOOP-9090.2.patch, HADOOP-9090.justEnhanceDefaultImpl.2.patch, HADOOP-9090.justEnhanceDefaultImpl.3.patch, HADOOP-9090.justEnhanceDefaultImpl.4.patch, HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch We have a need to publish metrics out of some short-living processes, which is not really well-suited to the current metrics system implementation which periodically publishes metrics asynchronously (a behavior that works great for long-living processes). Of course I could write my own metrics system, but it seems like such a waste to rewrite all the awesome code currently in the MetricsSystemImpl and supporting classes. The way I'm proposing to solve this is to: 1. Refactor the MetricsSystemImpl class into an abstract base MetricsSystemImpl class (common configuration and other code) and a concrete PeriodicPublishMetricsSystemImpl class (timer thread). 2. Refactor the MetricsSinkAdapter class into an abstract base MetricsSinkAdapter class (common configuration and other code) and a concrete AsyncMetricsSinkAdapter class (asynchronous publishing using the SinkQueue). 3. Derive a new simple class OnDemandPublishMetricsSystemImpl from MetricsSystemImpl, that just exposes a synchronous publish() method to do all the work. 4. Derive a SyncMetricsSinkAdapter class from MetricsSinkAdapter to just synchronously push metrics to the underlying sink. Does that sound reasonable? I'll attach the patch with all this coded up and simple tests (could use some polish I guess, but wanted to get everyone's opinion first). Notice that this is somewhat of a breaking change since MetricsSystemImpl is public (although it's marked with InterfaceAudience.Private); if the breaking change is a problem I could just rename the refactored classes so that PeriodicPublishMetricsSystemImpl is still called MetricsSystemImpl (and MetricsSystemImpl - BaseMetricsSystemImpl). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated HADOOP-8981: -- Attachment: HADOOP-8981-branch-trunk-win.2.patch Format the code TestMetricsSystemImpl fails on Windows -- Key: HADOOP-8981 URL: https://issues.apache.org/jira/browse/HADOOP-8981 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Arpit Agarwal Attachments: HADOOP-8981-branch-trunk-win.1.patch, HADOOP-8981-branch-trunk-win.2.patch The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (HADOOP-9103) when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some file whose file name has some special messy code(乱码)
[
https://issues.apache.org/jira/browse/HADOOP-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon moved HDFS-3307 to HADOOP-9103:
---
Component/s: (was: namenode)
io
Affects Version/s: (was: 0.20.1)
0.20.1
Key: HADOOP-9103 (was: HDFS-3307)
Project: Hadoop Common (was: Hadoop HDFS)
when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some
file whose file name has some special messy code(乱码)
-
Key: HADOOP-9103
URL: https://issues.apache.org/jira/browse/HADOOP-9103
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 0.20.1
Environment: SUSE LINUX
Reporter: yixiaohua
Assignee: Todd Lipcon
Attachments: FSImage.java, hadoop-9103.txt, ProblemString.txt,
TestUTF8AndStringGetBytes.java, TestUTF8AndStringGetBytes.java
Original Estimate: 12h
Remaining Estimate: 12h
this the log information of the exception from the SecondaryNameNode:
2012-03-28 00:48:42,553 ERROR
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
java.io.IOException: Found lease for
non-existent file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/@???
??tor.qzone.qq.com/keypart-00174
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
at java.lang.Thread.run(Thread.java:619)
this is the log information about the file from namenode:
2012-03-28 00:32:26,528 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=create
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 dst=null
perm=boss:boss:rw-r--r--
2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174.
blk_2751836614265659170_184668759
2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 is closed by
DFSClient_attempt_201203271849_0016_r_000174_0
2012-03-28 00:37:50,315 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=rename
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174
dst=/user/boss/pgv/fission/task16/split/ @?
tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
after check the code that save FSImage,I found there are a problem that maybe
a bug of HDFS Code,I past below:
-this is the saveFSImage method in FSImage.java, I make some
mark at the problem code
/**
* Save the contents of the FS image to the file.
*/
void saveFSImage(File newFile) throws IOException {
FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
FSDirectory fsDir = fsNamesys.dir;
long startTime = FSNamesystem.now();
//
// Write out data
//
DataOutputStream out = new DataOutputStream(
new BufferedOutputStream(
new
FileOutputStream(newFile)));
try {
.
// save the rest of the nodes
saveImage(strbuf, 0, fsDir.rootDir, out);--problem
fsNamesys.saveFilesUnderConstruction(out);--problem
detail is below
strbuf = null;
} finally {
out.close();
}
LOG.info(Image file of size + newFile.length() + saved in
+ (FSNamesystem.now() -
[jira] [Updated] (HADOOP-9103) UTF8 class does not properly decode Unicode characters outside the basic multilingual plane
[
https://issues.apache.org/jira/browse/HADOOP-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HADOOP-9103:
Summary: UTF8 class does not properly decode Unicode characters outside the
basic multilingual plane (was: when save FSImage ，HDFS（ or SecondaryNameNode
or FSImage）can't handle some file whose file name has some special messy
code(乱码))
UTF8 class does not properly decode Unicode characters outside the basic
multilingual plane
---
Key: HADOOP-9103
URL: https://issues.apache.org/jira/browse/HADOOP-9103
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 0.20.1
Environment: SUSE LINUX
Reporter: yixiaohua
Assignee: Todd Lipcon
Attachments: FSImage.java, hadoop-9103.txt, ProblemString.txt,
TestUTF8AndStringGetBytes.java, TestUTF8AndStringGetBytes.java
Original Estimate: 12h
Remaining Estimate: 12h
this the log information of the exception from the SecondaryNameNode:
2012-03-28 00:48:42,553 ERROR
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
java.io.IOException: Found lease for
non-existent file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/@???
??tor.qzone.qq.com/keypart-00174
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
at java.lang.Thread.run(Thread.java:619)
this is the log information about the file from namenode:
2012-03-28 00:32:26,528 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=create
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 dst=null
perm=boss:boss:rw-r--r--
2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174.
blk_2751836614265659170_184668759
2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 is closed by
DFSClient_attempt_201203271849_0016_r_000174_0
2012-03-28 00:37:50,315 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=rename
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174
dst=/user/boss/pgv/fission/task16/split/ @?
tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
after check the code that save FSImage,I found there are a problem that maybe
a bug of HDFS Code,I past below:
-this is the saveFSImage method in FSImage.java, I make some
mark at the problem code
/**
* Save the contents of the FS image to the file.
*/
void saveFSImage(File newFile) throws IOException {
FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
FSDirectory fsDir = fsNamesys.dir;
long startTime = FSNamesystem.now();
//
// Write out data
//
DataOutputStream out = new DataOutputStream(
new BufferedOutputStream(
new
FileOutputStream(newFile)));
try {
.
// save the rest of the nodes
saveImage(strbuf, 0, fsDir.rootDir, out);--problem
fsNamesys.saveFilesUnderConstruction(out);--problem
detail is below
strbuf = null;
} finally {
out.close();
}
LOG.info(Image file of size + newFile.length() + saved in
+ (FSNamesystem.now() - startTime)/1000 + seconds.);
}
/**
* Save file tree image starting from the given root.
* This is
[jira] [Updated] (HADOOP-9103) UTF8 class does not properly decode Unicode characters outside the basic multilingual plane
[
https://issues.apache.org/jira/browse/HADOOP-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HADOOP-9103:
Attachment: hadoop-9103.txt
Attached patch should fix this issue. I also amended the javadoc for UTF8 to
indicate that it encodes CESU-8
UTF8 class does not properly decode Unicode characters outside the basic
multilingual plane
---
Key: HADOOP-9103
URL: https://issues.apache.org/jira/browse/HADOOP-9103
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 0.20.1
Environment: SUSE LINUX
Reporter: yixiaohua
Assignee: Todd Lipcon
Attachments: FSImage.java, hadoop-9103.txt, ProblemString.txt,
TestUTF8AndStringGetBytes.java, TestUTF8AndStringGetBytes.java
Original Estimate: 12h
Remaining Estimate: 12h
this the log information of the exception from the SecondaryNameNode:
2012-03-28 00:48:42,553 ERROR
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
java.io.IOException: Found lease for
non-existent file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/@???
??tor.qzone.qq.com/keypart-00174
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
at java.lang.Thread.run(Thread.java:619)
this is the log information about the file from namenode:
2012-03-28 00:32:26,528 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=create
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 dst=null
perm=boss:boss:rw-r--r--
2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174.
blk_2751836614265659170_184668759
2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174 is closed by
DFSClient_attempt_201203271849_0016_r_000174_0
2012-03-28 00:37:50,315 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34cmd=rename
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@?tor.qzone.qq.com/keypart-00174
dst=/user/boss/pgv/fission/task16/split/ @?
tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
after check the code that save FSImage,I found there are a problem that maybe
a bug of HDFS Code,I past below:
-this is the saveFSImage method in FSImage.java, I make some
mark at the problem code
/**
* Save the contents of the FS image to the file.
*/
void saveFSImage(File newFile) throws IOException {
FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
FSDirectory fsDir = fsNamesys.dir;
long startTime = FSNamesystem.now();
//
// Write out data
//
DataOutputStream out = new DataOutputStream(
new BufferedOutputStream(
new
FileOutputStream(newFile)));
try {
.
// save the rest of the nodes
saveImage(strbuf, 0, fsDir.rootDir, out);--problem
fsNamesys.saveFilesUnderConstruction(out);--problem
detail is below
strbuf = null;
} finally {
out.close();
}
LOG.info(Image file of size + newFile.length() + saved in
+ (FSNamesystem.now() - startTime)/1000 + seconds.);
}
/**
* Save file tree image starting from the given root.
* This is a recursive procedure, which first saves all children of
* a current directory and then
