[jira] [Commented] (HADOOP-8722) -Dbundle.snappy doesn't work unless -Dsnappy.lib is set
[
https://issues.apache.org/jira/browse/HADOOP-8722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440096#comment-13440096
]
Colin Patrick McCabe commented on HADOOP-8722:
--
Looking at our code again, I think it requires the library to be dynamic, so I
can just skip the {{.a}} stuff. We use {{dlopen}}, and I don't think that
works with {{.a}} (static) libraries.
At that point, there no danger of making two copies of the {{.so}} file, since
the second copy would overwrite the first. If the developer puts two different
copies of libsnappy.so in {{$\{snappy.lib\}}} or {{$\{snappy.prefix\}}}, I
think we can safely discount that as user error.
-Dbundle.snappy doesn't work unless -Dsnappy.lib is set
---
Key: HADOOP-8722
URL: https://issues.apache.org/jira/browse/HADOOP-8722
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 2.2.0-alpha
Reporter: Eli Collins
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-8722.002.patch
HADOOP-8620 changed the default of snappy.lib from snappy.prefix/lib to
empty. This, in turn, means that you can't use {{-Dbundle.snappy}} without
setting {{-Dsnappy.lib}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8075) Lower native-hadoop library log from info to debug
[
https://issues.apache.org/jira/browse/HADOOP-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440104#comment-13440104
]
Hudson commented on HADOOP-8075:
Integrated in Hadoop-Mapreduce-trunk-Commit #2653 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2653/])
HADOOP-8075. Lower native-hadoop library log from info to debug.
Contributed by Hızır Sefa İrken (Revision 1376322)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376322
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java
Lower native-hadoop library log from info to debug
---
Key: HADOOP-8075
URL: https://issues.apache.org/jira/browse/HADOOP-8075
Project: Hadoop Common
Issue Type: Improvement
Components: native
Affects Versions: 0.23.0, 2.0.0-alpha
Reporter: Eli Collins
Assignee: Hızır Sefa İrken
Labels: newbie
Fix For: 2.2.0-alpha
Attachments: HDFS-8075.patch
The following log shows up in stderr all commands. We've already got a
warning if the native library can't be loaded, don't need to log this every
time at info level.
{noformat}
[eli@centos6 ~]$ hadoop fs -cat /user/eli/foo
12/02/12 20:10:20 INFO util.NativeCodeLoader: Loaded the native-hadoop library
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7682) taskTracker could not start because Failed to set permissions to ttprivate to 0700
[ https://issues.apache.org/jira/browse/HADOOP-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440120#comment-13440120 ] Joshua Caplan commented on HADOOP-7682: --- May I suggest a simple workaround for Windows users struggling with this new impediment. create a WinLocalFileSystem class (subclass of LocalFileSystem) which ignores IOExceptions on setPermissions() or, if you're feeling ambitious, does something more appropriate when trying to set them. You will also need to override mkdirs( Path, FsPermission ) to delegate to mkdirs( Path ) and call this.setPermission. Register this class as the implementation of the file schema, using the property definition fs.file.impl=com.package.name.WinLocalFileSystem That got me past it for local testing. taskTracker could not start because Failed to set permissions to ttprivate to 0700 -- Key: HADOOP-7682 URL: https://issues.apache.org/jira/browse/HADOOP-7682 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 1.0.1 Environment: OS:WindowsXP SP3 , Filesystem :NTFS, cygwin 1.7.9-1, jdk1.6.0_05 Reporter: Magic Xie ERROR org.apache.hadoop.mapred.TaskTracker:Can not start task tracker because java.io.IOException:Failed to set permissions of path:/tmp/hadoop-cyg_server/mapred/local/ttprivate to 0700 at org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:525) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:635) at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1328) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3430) Since hadoop0.20.203 when the TaskTracker initialize, it checks the permission(TaskTracker Line 624) of (org.apache.hadoop.mapred.TaskTracker.TT_LOG_TMP_DIR,org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR, org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR).RawLocalFileSystem(http://svn.apache.org/viewvc/hadoop/common/tags/release-0.20.203.0/src/core/org/apache/hadoop/fs/RawLocalFileSystem.java?view=markup) call setPermission(Line 481) to deal with it, setPermission works fine on *nx, however,it dose not alway works on windows. setPermission call setReadable of Java.io.File in the line 498, but according to the Table1 below provided by oracle,setReadable(false) will always return false on windows, the same as setExecutable(false). http://java.sun.com/developer/technicalArticles/J2SE/Desktop/javase6/enhancements/ is it cause the task tracker Failed to set permissions to ttprivate to 0700? Hadoop 0.20.202 works fine in the same environment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8031) Configuration class fails to find embedded .jar resources; should use URL.openStream()
[
https://issues.apache.org/jira/browse/HADOOP-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440122#comment-13440122
]
Ahmed Radwan commented on HADOOP-8031:
--
Thanks Elias,
I have noticed that the patch no longer applies due to the recent changes to
Configuration.java. I have made the required changes so it can apply cleanly on
trunk.
Also changed:
{code}
+doc = builder.parse((InputStream)name);
{code}
to:
{code}
+doc = parse(builder, (InputStream) resource);
{code}
to insure the InputStream will be closed.
Regarding testing, I am not sure if there is an easy way to reproduce this
issue in a unit test.
Configuration class fails to find embedded .jar resources; should use
URL.openStream()
--
Key: HADOOP-8031
URL: https://issues.apache.org/jira/browse/HADOOP-8031
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.0.0-alpha
Reporter: Elias Ross
Assignee: Elias Ross
Attachments: 0001-fix-HADOOP-7982-class-loader.patch,
HADOOP-8031.patch, hadoop-8031.txt
While running a hadoop client within RHQ (monitoring software) using its
classloader, I see this:
2012-02-07 09:15:25,313 INFO [ResourceContainer.invoker.daemon-2]
(org.apache.hadoop.conf.Configuration)- parsing
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
2012-02-07 09:15:25,318 ERROR [InventoryManager.discovery-1]
(rhq.core.pc.inventory.InventoryManager)- Failed to start component for
Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20,
name=NameNode, parent=vg61l01ad-hadoop002.apple.com] from synchronized merge.
org.rhq.core.clientapi.agent.PluginContainerException: Failed to start
component for resource Resource[id=16290, type=NameNode,
key=NameNode:/usr/lib/hadoop-0.20, name=NameNode,
parent=vg61l01ad-hadoop002.apple.com].
Caused by: java.lang.RuntimeException: core-site.xml not found
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1308)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1228)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1169)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)
This is because the URL
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
cannot be found by DocumentBuilder (doesn't understand it). (Note: the logs
are for an old version of Configuration class, but the new version has the
same code.)
The solution is to obtain the resource stream directly from the URL object
itself.
That is to say:
{code}
URL url = getResource((String)name);
-if (url != null) {
- if (!quiet) {
-LOG.info(parsing + url);
- }
- doc = builder.parse(url.toString());
-}
+doc = builder.parse(url.openStream());
{code}
Note: I have a full patch pending approval at Apple for this change,
including some cleanup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8031) Configuration class fails to find embedded .jar resources; should use URL.openStream()
[
https://issues.apache.org/jira/browse/HADOOP-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Radwan updated HADOOP-8031:
-
Attachment: HADOOP-8031.patch
Configuration class fails to find embedded .jar resources; should use
URL.openStream()
--
Key: HADOOP-8031
URL: https://issues.apache.org/jira/browse/HADOOP-8031
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.0.0-alpha
Reporter: Elias Ross
Assignee: Elias Ross
Attachments: 0001-fix-HADOOP-7982-class-loader.patch,
HADOOP-8031.patch, hadoop-8031.txt
While running a hadoop client within RHQ (monitoring software) using its
classloader, I see this:
2012-02-07 09:15:25,313 INFO [ResourceContainer.invoker.daemon-2]
(org.apache.hadoop.conf.Configuration)- parsing
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
2012-02-07 09:15:25,318 ERROR [InventoryManager.discovery-1]
(rhq.core.pc.inventory.InventoryManager)- Failed to start component for
Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20,
name=NameNode, parent=vg61l01ad-hadoop002.apple.com] from synchronized merge.
org.rhq.core.clientapi.agent.PluginContainerException: Failed to start
component for resource Resource[id=16290, type=NameNode,
key=NameNode:/usr/lib/hadoop-0.20, name=NameNode,
parent=vg61l01ad-hadoop002.apple.com].
Caused by: java.lang.RuntimeException: core-site.xml not found
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1308)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1228)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1169)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)
This is because the URL
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
cannot be found by DocumentBuilder (doesn't understand it). (Note: the logs
are for an old version of Configuration class, but the new version has the
same code.)
The solution is to obtain the resource stream directly from the URL object
itself.
That is to say:
{code}
URL url = getResource((String)name);
-if (url != null) {
- if (!quiet) {
-LOG.info(parsing + url);
- }
- doc = builder.parse(url.toString());
-}
+doc = builder.parse(url.openStream());
{code}
Note: I have a full patch pending approval at Apple for this change,
including some cleanup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8031) Configuration class fails to find embedded .jar resources; should use URL.openStream()
[
https://issues.apache.org/jira/browse/HADOOP-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Radwan updated HADOOP-8031:
-
Status: Patch Available (was: Open)
Configuration class fails to find embedded .jar resources; should use
URL.openStream()
--
Key: HADOOP-8031
URL: https://issues.apache.org/jira/browse/HADOOP-8031
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.0.0-alpha
Reporter: Elias Ross
Assignee: Elias Ross
Attachments: 0001-fix-HADOOP-7982-class-loader.patch,
HADOOP-8031.patch, hadoop-8031.txt
While running a hadoop client within RHQ (monitoring software) using its
classloader, I see this:
2012-02-07 09:15:25,313 INFO [ResourceContainer.invoker.daemon-2]
(org.apache.hadoop.conf.Configuration)- parsing
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
2012-02-07 09:15:25,318 ERROR [InventoryManager.discovery-1]
(rhq.core.pc.inventory.InventoryManager)- Failed to start component for
Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20,
name=NameNode, parent=vg61l01ad-hadoop002.apple.com] from synchronized merge.
org.rhq.core.clientapi.agent.PluginContainerException: Failed to start
component for resource Resource[id=16290, type=NameNode,
key=NameNode:/usr/lib/hadoop-0.20, name=NameNode,
parent=vg61l01ad-hadoop002.apple.com].
Caused by: java.lang.RuntimeException: core-site.xml not found
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1308)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1228)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1169)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)
This is because the URL
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
cannot be found by DocumentBuilder (doesn't understand it). (Note: the logs
are for an old version of Configuration class, but the new version has the
same code.)
The solution is to obtain the resource stream directly from the URL object
itself.
That is to say:
{code}
URL url = getResource((String)name);
-if (url != null) {
- if (!quiet) {
-LOG.info(parsing + url);
- }
- doc = builder.parse(url.toString());
-}
+doc = builder.parse(url.openStream());
{code}
Note: I have a full patch pending approval at Apple for this change,
including some cleanup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8632) Configuration leaking class-loaders
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Costin Leau updated HADOOP-8632: Attachment: HADOOP-8632-trunk-no-tabs.patch the patch w/o any tabs. Configuration leaking class-loaders --- Key: HADOOP-8632 URL: https://issues.apache.org/jira/browse/HADOOP-8632 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 2.0.0-alpha Reporter: Costin Leau Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk-no-tabs.patch, HADOOP-8632-trunk.patch The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed. One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything. In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself. Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed. And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8075) Lower native-hadoop library log from info to debug
[
https://issues.apache.org/jira/browse/HADOOP-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440141#comment-13440141
]
Hudson commented on HADOOP-8075:
Integrated in Hadoop-Hdfs-trunk-Commit #2689 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2689/])
HADOOP-8075. Lower native-hadoop library log from info to debug.
Contributed by Hızır Sefa İrken (Revision 1376322)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376322
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java
Lower native-hadoop library log from info to debug
---
Key: HADOOP-8075
URL: https://issues.apache.org/jira/browse/HADOOP-8075
Project: Hadoop Common
Issue Type: Improvement
Components: native
Affects Versions: 0.23.0, 2.0.0-alpha
Reporter: Eli Collins
Assignee: Hızır Sefa İrken
Labels: newbie
Fix For: 2.2.0-alpha
Attachments: HDFS-8075.patch
The following log shows up in stderr all commands. We've already got a
warning if the native library can't be loaded, don't need to log this every
time at info level.
{noformat}
[eli@centos6 ~]$ hadoop fs -cat /user/eli/foo
12/02/12 20:10:20 INFO util.NativeCodeLoader: Loaded the native-hadoop library
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8722) -Dbundle.snappy doesn't work unless -Dsnappy.lib is set
[
https://issues.apache.org/jira/browse/HADOOP-8722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440143#comment-13440143
]
Hadoop QA commented on HADOOP-8722:
---
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542061/HADOOP-8722.002.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+0 tests included. The patch appears to be a documentation patch that
doesn't require tests.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1349//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1349//console
This message is automatically generated.
-Dbundle.snappy doesn't work unless -Dsnappy.lib is set
---
Key: HADOOP-8722
URL: https://issues.apache.org/jira/browse/HADOOP-8722
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 2.2.0-alpha
Reporter: Eli Collins
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-8722.002.patch
HADOOP-8620 changed the default of snappy.lib from snappy.prefix/lib to
empty. This, in turn, means that you can't use {{-Dbundle.snappy}} without
setting {{-Dsnappy.lib}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8075) Lower native-hadoop library log from info to debug
[
https://issues.apache.org/jira/browse/HADOOP-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440159#comment-13440159
]
Hudson commented on HADOOP-8075:
Integrated in Hadoop-Common-trunk-Commit #2625 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2625/])
HADOOP-8075. Lower native-hadoop library log from info to debug.
Contributed by Hızır Sefa İrken (Revision 1376322)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376322
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java
Lower native-hadoop library log from info to debug
---
Key: HADOOP-8075
URL: https://issues.apache.org/jira/browse/HADOOP-8075
Project: Hadoop Common
Issue Type: Improvement
Components: native
Affects Versions: 0.23.0, 2.0.0-alpha
Reporter: Eli Collins
Assignee: Hızır Sefa İrken
Labels: newbie
Fix For: 2.2.0-alpha
Attachments: HDFS-8075.patch
The following log shows up in stderr all commands. We've already got a
warning if the native library can't be loaded, don't need to log this every
time at info level.
{noformat}
[eli@centos6 ~]$ hadoop fs -cat /user/eli/foo
12/02/12 20:10:20 INFO util.NativeCodeLoader: Loaded the native-hadoop library
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8655) In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character s
[ https://issues.apache.org/jira/browse/HADOOP-8655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440166#comment-13440166 ] Hadoop QA commented on HADOOP-8655: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542076/HADOOP-8655%20%282%29.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1348//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1348//console This message is automatically generated. In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character sequence in delimiter were found missing in certain cases in the Map Output - Key: HADOOP-8655 URL: https://issues.apache.org/jira/browse/HADOOP-8655 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 0.20.2 Environment: Linux- Ubuntu 10.04 Reporter: Arun A K Labels: hadoop, mapreduce, textinputformat, textinputformat.record.delimiter Attachments: HADOOP-8655 (2).patch, HADOOP-8655.patch, HADOOP-8655.patch, HADOOP-8655.patch, MAPREDUCE-4519.patch Original Estimate: 168h Remaining Estimate: 168h Set textinputformat.record.delimiter as /entity Suppose the input is a text file with the following content entityid1/idnameUser1/name/entityentityid2/idnameUser2/name/entityentityid3/idnameUser3/name/entityentityid4/idnameUser4/name/entityentityid5/idnameUser5/name/entity Mapper was expected to get value as Value 1 - entityid1/idnameUser1/name Value 2 - entityid2/idnameUser2/name Value 3 - entityid3/idnameUser3/name Value 4 - entityid4/idnameUser4/name Value 5 - entityid5/idnameUser5/name According to this bug Mapper gets value Value 1 - entityid1/idnameUser1/name Value 2 - entityid2/idnameUser2/name Value 3 - entityid3idnameUser3/name Value 4 - entityid4/idnameUser4name Value 5 - entityid5/idnameUser5/name The pattern shown above need not occur for value 1,2,3 necessarily. The bug occurs at some random positions in the map input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8031) Configuration class fails to find embedded .jar resources; should use URL.openStream()
[
https://issues.apache.org/jira/browse/HADOOP-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440169#comment-13440169
]
Hadoop QA commented on HADOOP-8031:
---
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542098/HADOOP-8031.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in
hadoop-common-project/hadoop-common.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1350//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1350//console
This message is automatically generated.
Configuration class fails to find embedded .jar resources; should use
URL.openStream()
--
Key: HADOOP-8031
URL: https://issues.apache.org/jira/browse/HADOOP-8031
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.0.0-alpha
Reporter: Elias Ross
Assignee: Elias Ross
Attachments: 0001-fix-HADOOP-7982-class-loader.patch,
HADOOP-8031.patch, hadoop-8031.txt
While running a hadoop client within RHQ (monitoring software) using its
classloader, I see this:
2012-02-07 09:15:25,313 INFO [ResourceContainer.invoker.daemon-2]
(org.apache.hadoop.conf.Configuration)- parsing
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
2012-02-07 09:15:25,318 ERROR [InventoryManager.discovery-1]
(rhq.core.pc.inventory.InventoryManager)- Failed to start component for
Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20,
name=NameNode, parent=vg61l01ad-hadoop002.apple.com] from synchronized merge.
org.rhq.core.clientapi.agent.PluginContainerException: Failed to start
component for resource Resource[id=16290, type=NameNode,
key=NameNode:/usr/lib/hadoop-0.20, name=NameNode,
parent=vg61l01ad-hadoop002.apple.com].
Caused by: java.lang.RuntimeException: core-site.xml not found
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1308)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1228)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1169)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)
This is because the URL
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
cannot be found by DocumentBuilder (doesn't understand it). (Note: the logs
are for an old version of Configuration class, but the new version has the
same code.)
The solution is to obtain the resource stream directly from the URL object
itself.
That is to say:
{code}
URL url = getResource((String)name);
-if (url != null) {
- if (!quiet) {
-LOG.info(parsing + url);
- }
- doc = builder.parse(url.toString());
-}
+doc = builder.parse(url.openStream());
{code}
Note: I have a full patch pending approval at Apple for this change,
including some cleanup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking class-loaders
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440172#comment-13440172 ] Hadoop QA commented on HADOOP-8632: --- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542100/HADOOP-8632-trunk-no-tabs.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1351//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1351//console This message is automatically generated. Configuration leaking class-loaders --- Key: HADOOP-8632 URL: https://issues.apache.org/jira/browse/HADOOP-8632 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 2.0.0-alpha Reporter: Costin Leau Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk-no-tabs.patch, HADOOP-8632-trunk.patch The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed. One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything. In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself. Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed. And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8720) TestLocalFileSystem should use test root subdirectory
[ https://issues.apache.org/jira/browse/HADOOP-8720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440254#comment-13440254 ] Hudson commented on HADOOP-8720: Integrated in Hadoop-Hdfs-trunk #1143 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1143/]) HADOOP-8720. TestLocalFileSystem should use test root subdirectory. Contributed by Vlad Rozov (Revision 1376123) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376123 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java TestLocalFileSystem should use test root subdirectory - Key: HADOOP-8720 URL: https://issues.apache.org/jira/browse/HADOOP-8720 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 2.0.0-alpha Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Trivial Fix For: 2.2.0-alpha Attachments: HADOOP-8720.patch During unit test root directory of test.build.data is deleted possibly affecting other tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8712) Change default hadoop.security.group.mapping
[ https://issues.apache.org/jira/browse/HADOOP-8712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated HADOOP-8712: -- Affects Version/s: (was: 0.23.3) Change default hadoop.security.group.mapping Key: HADOOP-8712 URL: https://issues.apache.org/jira/browse/HADOOP-8712 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.2.0-alpha Reporter: Robert Parker Assignee: Robert Parker Priority: Minor Change the hadoop.security.group.mapping in core-site to JniBasedUnixGroupsNetgroupMappingWithFallback -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8712) Change default hadoop.security.group.mapping
[ https://issues.apache.org/jira/browse/HADOOP-8712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated HADOOP-8712: -- Affects Version/s: (was: 2.0.0-alpha) 2.2.0-alpha Change default hadoop.security.group.mapping Key: HADOOP-8712 URL: https://issues.apache.org/jira/browse/HADOOP-8712 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.2.0-alpha Reporter: Robert Parker Assignee: Robert Parker Priority: Minor Change the hadoop.security.group.mapping in core-site to JniBasedUnixGroupsNetgroupMappingWithFallback -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8712) Change default hadoop.security.group.mapping
[ https://issues.apache.org/jira/browse/HADOOP-8712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated HADOOP-8712: -- Attachment: HADOOP-8712-v1.patch Changed the default to JniBasedUnixGroupsMappingWithFallback. Updated the documentation. Change default hadoop.security.group.mapping Key: HADOOP-8712 URL: https://issues.apache.org/jira/browse/HADOOP-8712 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.2.0-alpha Reporter: Robert Parker Assignee: Robert Parker Priority: Minor Attachments: HADOOP-8712-v1.patch Change the hadoop.security.group.mapping in core-site to JniBasedUnixGroupsNetgroupMappingWithFallback -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8712) Change default hadoop.security.group.mapping
[ https://issues.apache.org/jira/browse/HADOOP-8712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated HADOOP-8712: -- Status: Patch Available (was: Open) Change default hadoop.security.group.mapping Key: HADOOP-8712 URL: https://issues.apache.org/jira/browse/HADOOP-8712 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.2.0-alpha Reporter: Robert Parker Assignee: Robert Parker Priority: Minor Attachments: HADOOP-8712-v1.patch Change the hadoop.security.group.mapping in core-site to JniBasedUnixGroupsNetgroupMappingWithFallback -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8225) DistCp fails when invoked by Oozie
[
https://issues.apache.org/jira/browse/HADOOP-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HADOOP-8225:
Attachment: HADOOP-8225.patch
Add a bunch of tests. Corrected {{NamedToken}}'s equality impl.
DistCp fails when invoked by Oozie
--
Key: HADOOP-8225
URL: https://issues.apache.org/jira/browse/HADOOP-8225
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.23.1
Reporter: Mithun Radhakrishnan
Assignee: Daryn Sharp
Attachments: HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch,
HADOOP-8225.patch, HADOOP-8225.patch
When DistCp is invoked through a proxy-user (e.g. through Oozie), the
delegation-token-store isn't picked up by DistCp correctly. One sees failures
such as:
ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp
operation:
java.lang.SecurityException: Intercepted System.exit(-999)
at
org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651)
at java.lang.Runtime.exit(Runtime.java:88)
at java.lang.System.exit(System.java:904)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:357)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't
being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post
a patch for this shortly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8632) Configuration leaking class-loaders
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HADOOP-8632: Resolution: Fixed Fix Version/s: 2.2.0-alpha 3.0.0 Status: Resolved (was: Patch Available) Thanks Constin, I merged this into trunk, and branch-2. If you wanted it in another release please add a comment to indicate it. Configuration leaking class-loaders --- Key: HADOOP-8632 URL: https://issues.apache.org/jira/browse/HADOOP-8632 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 2.0.0-alpha Reporter: Costin Leau Fix For: 3.0.0, 2.2.0-alpha Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk-no-tabs.patch, HADOOP-8632-trunk.patch The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed. One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything. In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself. Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed. And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8632) Configuration leaking class-loaders
[ https://issues.apache.org/jira/browse/HADOOP-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440391#comment-13440391 ] Hudson commented on HADOOP-8632: Integrated in Hadoop-Mapreduce-trunk-Commit #2654 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2654/]) HADOOP-8632. Configuration leaking class-loaders (Costin Leau via bobby) (Revision 1376543) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376543 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java Configuration leaking class-loaders --- Key: HADOOP-8632 URL: https://issues.apache.org/jira/browse/HADOOP-8632 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 2.0.0-alpha Reporter: Costin Leau Fix For: 3.0.0, 2.2.0-alpha Attachments: 0001-wrapping-classes-with-WeakRefs-in-CLASS_CACHE.patch, HADOOP-8632.patch, HADOOP-8632-trunk-no-tabs.patch, HADOOP-8632-trunk.patch The newly introduced CACHE_CLASSES leaks class loaders causing associated classes to not be reclaimed. One solution is to remove the cache itself since each class loader implementation caches the classes it loads automatically and preventing an exception from being raised is just a micro-optimization that, as one can tell, causes bugs instead of improving anything. In fact, I would argue in a highly-concurrent environment, the weakhashmap synchronization/lookup probably costs more then creating the exception itself. Another is to prevent the leak from occurring, by inserting the loadedclass into the WeakHashMap wrapped in a WeakReference. Otherwise the class has a strong reference to its classloader (the key) meaning neither gets GC'ed. And since the cache_class is static, even if the originating Configuration instance gets GC'ed, its classloader won't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8225) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/HADOOP-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-8225: Component/s: security Priority: Blocker (was: Major) Target Version/s: 0.23.3, 3.0.0, 2.2.0-alpha Affects Version/s: 3.0.0 2.0.0-alpha Marking as blocker since distcp and other apps that launch jobs can't work under oozie. DistCp fails when invoked by Oozie -- Key: HADOOP-8225 URL: https://issues.apache.org/jira/browse/HADOOP-8225 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.1, 2.0.0-alpha, 3.0.0 Reporter: Mithun Radhakrishnan Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8709) globStatus changed behavior from 0.20/1.x
[ https://issues.apache.org/jira/browse/HADOOP-8709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440396#comment-13440396 ] Robert Joseph Evans commented on HADOOP-8709: - If we do want different behavior we should add in a new API and deprecate the old one. This is how most projects evolve while still maintaining backwards compatibility. I like Chris's idea for the new API, and I would actually go a step further and create an Glob class, that holds the base Path and the regexp/pattern. That way other methods that may take a glob, like in Shell or Util, it is obvious when something will be interpreted as a Glob and when it will be interpreted as a Path. But all of that can and should be done in a separate JIRA. I agree with Sanjay also. This new API should remove the null return values. It will through a FNFE when the base path is not found, and will return an empty iterator when the pattern does not match anything. I think this solution would address all concerns. I am happy to file the separate JIRA for the updated API and I owuld like to check in the current patch, assuming that there are no objections to this. globStatus changed behavior from 0.20/1.x - Key: HADOOP-8709 URL: https://issues.apache.org/jira/browse/HADOOP-8709 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.23.0, 2.0.0-alpha Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: HADOOP-8709.patch, HADOOP-8709.patch In 0.20 or 1.x, globStatus will return an empty array if the glob pattern does not match any files. After HADOOP-6201 it throws FileNotFoundException. The javadoc states it will return an empty array. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8225) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/HADOOP-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440400#comment-13440400 ] Hadoop QA commented on HADOOP-8225: --- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542139/HADOOP-8225.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-distcp. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1353//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1353//console This message is automatically generated. DistCp fails when invoked by Oozie -- Key: HADOOP-8225 URL: https://issues.apache.org/jira/browse/HADOOP-8225 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.1, 2.0.0-alpha, 3.0.0 Reporter: Mithun Radhakrishnan Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8655) In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character s
[ https://issues.apache.org/jira/browse/HADOOP-8655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440413#comment-13440413 ] Arun A K commented on HADOOP-8655: -- Gelesh :I had tried out the patch that you have posted herein. That really solves my problem. Thanks a lot for the patch. Is the patch that you re uploaded same as before? Do I need to apply this new patch? In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character sequence in delimiter were found missing in certain cases in the Map Output - Key: HADOOP-8655 URL: https://issues.apache.org/jira/browse/HADOOP-8655 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 0.20.2 Environment: Linux- Ubuntu 10.04 Reporter: Arun A K Labels: hadoop, mapreduce, textinputformat, textinputformat.record.delimiter Attachments: HADOOP-8655 (2).patch, HADOOP-8655.patch, HADOOP-8655.patch, HADOOP-8655.patch, MAPREDUCE-4519.patch Original Estimate: 168h Remaining Estimate: 168h Set textinputformat.record.delimiter as /entity Suppose the input is a text file with the following content entityid1/idnameUser1/name/entityentityid2/idnameUser2/name/entityentityid3/idnameUser3/name/entityentityid4/idnameUser4/name/entityentityid5/idnameUser5/name/entity Mapper was expected to get value as Value 1 - entityid1/idnameUser1/name Value 2 - entityid2/idnameUser2/name Value 3 - entityid3/idnameUser3/name Value 4 - entityid4/idnameUser4/name Value 5 - entityid5/idnameUser5/name According to this bug Mapper gets value Value 1 - entityid1/idnameUser1/name Value 2 - entityid2/idnameUser2/name Value 3 - entityid3idnameUser3/name Value 4 - entityid4/idnameUser4name Value 5 - entityid5/idnameUser5/name The pattern shown above need not occur for value 1,2,3 necessarily. The bug occurs at some random positions in the map input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8225) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/HADOOP-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440416#comment-13440416 ] Robert Joseph Evans commented on HADOOP-8225: - I have done a quick look thorough the patch and it looks good to me. I am holding off on a full +1 until I have looked more deeply into the patch as I am not really a token expert. If someone else gives it a +1 feel free to check it in without waiting for me. DistCp fails when invoked by Oozie -- Key: HADOOP-8225 URL: https://issues.apache.org/jira/browse/HADOOP-8225 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.1, 2.0.0-alpha, 3.0.0 Reporter: Mithun Radhakrishnan Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8723) Remove tests and tests-sources jars from classpath
Jason Lowe created HADOOP-8723:
--
Summary: Remove tests and tests-sources jars from classpath
Key: HADOOP-8723
URL: https://issues.apache.org/jira/browse/HADOOP-8723
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Jason Lowe
Currently {{hadoop-config.sh}} is including any tests and tests-sources jars in
the default classpath, as those jars are shipped in the dist tarball next to
the main jars and the script is globbing everything in that directory.
The tests and tests-sources jars aren't required to run Hadoop, but they can
cause breakage when inadvertently picked up. See HDFS-3831 as an example.
Ideally we should not be adding these jars to the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8655) In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character seq
[ https://issues.apache.org/jira/browse/HADOOP-8655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HADOOP-8655: Resolution: Fixed Fix Version/s: 2.2.0-alpha 3.0.0 Status: Resolved (was: Patch Available) Thanks for the patch Gelesh +1. I checked this into trunk and branch-2. In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character sequence in delimiter were found missing in certain cases in the Map Output - Key: HADOOP-8655 URL: https://issues.apache.org/jira/browse/HADOOP-8655 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 0.20.2 Environment: Linux- Ubuntu 10.04 Reporter: Arun A K Labels: hadoop, mapreduce, textinputformat, textinputformat.record.delimiter Fix For: 3.0.0, 2.2.0-alpha Attachments: HADOOP-8655 (2).patch, HADOOP-8655.patch, HADOOP-8655.patch, HADOOP-8655.patch, MAPREDUCE-4519.patch Original Estimate: 168h Remaining Estimate: 168h Set textinputformat.record.delimiter as /entity Suppose the input is a text file with the following content entityid1/idnameUser1/name/entityentityid2/idnameUser2/name/entityentityid3/idnameUser3/name/entityentityid4/idnameUser4/name/entityentityid5/idnameUser5/name/entity Mapper was expected to get value as Value 1 - entityid1/idnameUser1/name Value 2 - entityid2/idnameUser2/name Value 3 - entityid3/idnameUser3/name Value 4 - entityid4/idnameUser4/name Value 5 - entityid5/idnameUser5/name According to this bug Mapper gets value Value 1 - entityid1/idnameUser1/name Value 2 - entityid2/idnameUser2/name Value 3 - entityid3idnameUser3/name Value 4 - entityid4/idnameUser4name Value 5 - entityid5/idnameUser5/name The pattern shown above need not occur for value 1,2,3 necessarily. The bug occurs at some random positions in the map input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8712) Change default hadoop.security.group.mapping
[ https://issues.apache.org/jira/browse/HADOOP-8712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440450#comment-13440450 ] Hadoop QA commented on HADOOP-8712: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542136/HADOOP-8712-v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.hdfs.TestReplication +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1352//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1352//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1352//console This message is automatically generated. Change default hadoop.security.group.mapping Key: HADOOP-8712 URL: https://issues.apache.org/jira/browse/HADOOP-8712 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.2.0-alpha Reporter: Robert Parker Assignee: Robert Parker Priority: Minor Attachments: HADOOP-8712-v1.patch Change the hadoop.security.group.mapping in core-site to JniBasedUnixGroupsNetgroupMappingWithFallback -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8655) In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character s
[ https://issues.apache.org/jira/browse/HADOOP-8655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440452#comment-13440452 ] Hudson commented on HADOOP-8655: Integrated in Hadoop-Common-trunk-Commit #2627 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2627/]) HADOOP-8655. Fix TextInputFormat for large deliminators. (Gelesh via bobby) (Revision 1376592) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376592 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character sequence in delimiter were found missing in certain cases in the Map Output - Key: HADOOP-8655 URL: https://issues.apache.org/jira/browse/HADOOP-8655 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 0.20.2 Environment: Linux- Ubuntu 10.04 Reporter: Arun A K Labels: hadoop, mapreduce, textinputformat, textinputformat.record.delimiter Fix For: 3.0.0, 2.2.0-alpha Attachments: HADOOP-8655 (2).patch, HADOOP-8655.patch, HADOOP-8655.patch, HADOOP-8655.patch, MAPREDUCE-4519.patch Original Estimate: 168h Remaining Estimate: 168h Set textinputformat.record.delimiter as /entity Suppose the input is a text file with the following content entityid1/idnameUser1/name/entityentityid2/idnameUser2/name/entityentityid3/idnameUser3/name/entityentityid4/idnameUser4/name/entityentityid5/idnameUser5/name/entity Mapper was expected to get value as Value 1 - entityid1/idnameUser1/name Value 2 - entityid2/idnameUser2/name Value 3 - entityid3/idnameUser3/name Value 4 - entityid4/idnameUser4/name Value 5 - entityid5/idnameUser5/name According to this bug Mapper gets value Value 1 - entityid1/idnameUser1/name Value 2 - entityid2/idnameUser2/name Value 3 - entityid3idnameUser3/name Value 4 - entityid4/idnameUser4name Value 5 - entityid5/idnameUser5/name The pattern shown above need not occur for value 1,2,3 necessarily. The bug occurs at some random positions in the map input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8655) In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character s
[ https://issues.apache.org/jira/browse/HADOOP-8655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440453#comment-13440453 ] Hudson commented on HADOOP-8655: Integrated in Hadoop-Hdfs-trunk-Commit #2691 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2691/]) HADOOP-8655. Fix TextInputFormat for large deliminators. (Gelesh via bobby) (Revision 1376592) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376592 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character sequence in delimiter were found missing in certain cases in the Map Output - Key: HADOOP-8655 URL: https://issues.apache.org/jira/browse/HADOOP-8655 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 0.20.2 Environment: Linux- Ubuntu 10.04 Reporter: Arun A K Labels: hadoop, mapreduce, textinputformat, textinputformat.record.delimiter Fix For: 3.0.0, 2.2.0-alpha Attachments: HADOOP-8655 (2).patch, HADOOP-8655.patch, HADOOP-8655.patch, HADOOP-8655.patch, MAPREDUCE-4519.patch Original Estimate: 168h Remaining Estimate: 168h Set textinputformat.record.delimiter as /entity Suppose the input is a text file with the following content entityid1/idnameUser1/name/entityentityid2/idnameUser2/name/entityentityid3/idnameUser3/name/entityentityid4/idnameUser4/name/entityentityid5/idnameUser5/name/entity Mapper was expected to get value as Value 1 - entityid1/idnameUser1/name Value 2 - entityid2/idnameUser2/name Value 3 - entityid3/idnameUser3/name Value 4 - entityid4/idnameUser4/name Value 5 - entityid5/idnameUser5/name According to this bug Mapper gets value Value 1 - entityid1/idnameUser1/name Value 2 - entityid2/idnameUser2/name Value 3 - entityid3idnameUser3/name Value 4 - entityid4/idnameUser4name Value 5 - entityid5/idnameUser5/name The pattern shown above need not occur for value 1,2,3 necessarily. The bug occurs at some random positions in the map input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8655) In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character s
[ https://issues.apache.org/jira/browse/HADOOP-8655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440455#comment-13440455 ] Hudson commented on HADOOP-8655: Integrated in Hadoop-Mapreduce-trunk-Commit #2655 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2655/]) HADOOP-8655. Fix TextInputFormat for large deliminators. (Gelesh via bobby) (Revision 1376592) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376592 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestLineReader.java In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character sequence in delimiter were found missing in certain cases in the Map Output - Key: HADOOP-8655 URL: https://issues.apache.org/jira/browse/HADOOP-8655 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 0.20.2 Environment: Linux- Ubuntu 10.04 Reporter: Arun A K Labels: hadoop, mapreduce, textinputformat, textinputformat.record.delimiter Fix For: 3.0.0, 2.2.0-alpha Attachments: HADOOP-8655 (2).patch, HADOOP-8655.patch, HADOOP-8655.patch, HADOOP-8655.patch, MAPREDUCE-4519.patch Original Estimate: 168h Remaining Estimate: 168h Set textinputformat.record.delimiter as /entity Suppose the input is a text file with the following content entityid1/idnameUser1/name/entityentityid2/idnameUser2/name/entityentityid3/idnameUser3/name/entityentityid4/idnameUser4/name/entityentityid5/idnameUser5/name/entity Mapper was expected to get value as Value 1 - entityid1/idnameUser1/name Value 2 - entityid2/idnameUser2/name Value 3 - entityid3/idnameUser3/name Value 4 - entityid4/idnameUser4/name Value 5 - entityid5/idnameUser5/name According to this bug Mapper gets value Value 1 - entityid1/idnameUser1/name Value 2 - entityid2/idnameUser2/name Value 3 - entityid3idnameUser3/name Value 4 - entityid4/idnameUser4name Value 5 - entityid5/idnameUser5/name The pattern shown above need not occur for value 1,2,3 necessarily. The bug occurs at some random positions in the map input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8225) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/HADOOP-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440459#comment-13440459 ] Devaraj Das commented on HADOOP-8225: - Will take a look. DistCp fails when invoked by Oozie -- Key: HADOOP-8225 URL: https://issues.apache.org/jira/browse/HADOOP-8225 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.1, 2.0.0-alpha, 3.0.0 Reporter: Mithun Radhakrishnan Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8723) Remove tests and tests-sources jars from classpath
[
https://issues.apache.org/jira/browse/HADOOP-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated HADOOP-8723:
---
Attachment: HADOOP-8723.patch
Patch to modify the dist tarball so all the tests and tests-sources jars are in
a test/ subdirectory in their respective project-specific areas.
Manually tested by bringing up a single-node cluster and running teragen,
terasort, wordcount, and sleep jobs. Note that after this change the sleep
jobs need the classpath modified, e.g.: via {{HADOOP_CLASSPATH}}, to add in
{{$HADOOP_PREFIX/share/hadoop/common/test/*.jar}} as it relies on test classes
in common.
We may need to mark this as an incompatible change for those who were relying
on the test classes in the path for their own tests. I also noticed that in my
single-node cluster setup I was inadvertently picking up {{log4j.properties}}
from one of the tests jars before this change, and I needed to provide a
properties file after this change to restore logging.
Remove tests and tests-sources jars from classpath
--
Key: HADOOP-8723
URL: https://issues.apache.org/jira/browse/HADOOP-8723
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Jason Lowe
Attachments: HADOOP-8723.patch
Currently {{hadoop-config.sh}} is including any tests and tests-sources jars
in the default classpath, as those jars are shipped in the dist tarball next
to the main jars and the script is globbing everything in that directory.
The tests and tests-sources jars aren't required to run Hadoop, but they can
cause breakage when inadvertently picked up. See HDFS-3831 as an example.
Ideally we should not be adding these jars to the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8723) Remove tests and tests-sources jars from classpath
[
https://issues.apache.org/jira/browse/HADOOP-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated HADOOP-8723:
---
Assignee: Jason Lowe
Target Version/s: 3.0.0, 2.2.0-alpha
Status: Patch Available (was: Open)
Remove tests and tests-sources jars from classpath
--
Key: HADOOP-8723
URL: https://issues.apache.org/jira/browse/HADOOP-8723
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: HADOOP-8723.patch
Currently {{hadoop-config.sh}} is including any tests and tests-sources jars
in the default classpath, as those jars are shipped in the dist tarball next
to the main jars and the script is globbing everything in that directory.
The tests and tests-sources jars aren't required to run Hadoop, but they can
cause breakage when inadvertently picked up. See HDFS-3831 as an example.
Ideally we should not be adding these jars to the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8723) Remove tests and tests-sources jars from classpath
[
https://issues.apache.org/jira/browse/HADOOP-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440487#comment-13440487
]
Eli Collins commented on HADOOP-8723:
-
How about modifying the sleep job (and any other similar jobs) so they don't
depend on the test classes?
Remove tests and tests-sources jars from classpath
--
Key: HADOOP-8723
URL: https://issues.apache.org/jira/browse/HADOOP-8723
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: HADOOP-8723.patch
Currently {{hadoop-config.sh}} is including any tests and tests-sources jars
in the default classpath, as those jars are shipped in the dist tarball next
to the main jars and the script is globbing everything in that directory.
The tests and tests-sources jars aren't required to run Hadoop, but they can
cause breakage when inadvertently picked up. See HDFS-3831 as an example.
Ideally we should not be adding these jars to the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8225) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/HADOOP-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440493#comment-13440493 ] Devaraj Das commented on HADOOP-8225: - The patch looks good on the tokens side. +1. DistCp fails when invoked by Oozie -- Key: HADOOP-8225 URL: https://issues.apache.org/jira/browse/HADOOP-8225 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.1, 2.0.0-alpha, 3.0.0 Reporter: Mithun Radhakrishnan Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8723) Remove tests and tests-sources jars from classpath
[
https://issues.apache.org/jira/browse/HADOOP-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440498#comment-13440498
]
Jason Lowe commented on HADOOP-8723:
Sounds good. Maybe make sleep job a full-fledged example since it seems to be
used quite a bit? Part of this JIRA, or better as a pre-requisite?
Remove tests and tests-sources jars from classpath
--
Key: HADOOP-8723
URL: https://issues.apache.org/jira/browse/HADOOP-8723
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: HADOOP-8723.patch
Currently {{hadoop-config.sh}} is including any tests and tests-sources jars
in the default classpath, as those jars are shipped in the dist tarball next
to the main jars and the script is globbing everything in that directory.
The tests and tests-sources jars aren't required to run Hadoop, but they can
cause breakage when inadvertently picked up. See HDFS-3831 as an example.
Ideally we should not be adding these jars to the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8225) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/HADOOP-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440500#comment-13440500 ] Robert Joseph Evans commented on HADOOP-8225: - I have spent some time digging into this too and I feel confident that it works. +1 I'll check this in. DistCp fails when invoked by Oozie -- Key: HADOOP-8225 URL: https://issues.apache.org/jira/browse/HADOOP-8225 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.1, 2.0.0-alpha, 3.0.0 Reporter: Mithun Radhakrishnan Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8225) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/HADOOP-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HADOOP-8225: Resolution: Fixed Fix Version/s: 2.2.0-alpha 3.0.0 2.1.0-alpha 0.23.3 Status: Resolved (was: Patch Available) Thanks Daryn, I put this into trunk, branch-2, branch-2.1.0-alpha, and branch-0.23 DistCp fails when invoked by Oozie -- Key: HADOOP-8225 URL: https://issues.apache.org/jira/browse/HADOOP-8225 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.1, 2.0.0-alpha, 3.0.0 Reporter: Mithun Radhakrishnan Assignee: Daryn Sharp Priority: Blocker Fix For: 0.23.3, 2.1.0-alpha, 3.0.0, 2.2.0-alpha Attachments: HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8723) Remove tests and tests-sources jars from classpath
[
https://issues.apache.org/jira/browse/HADOOP-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440511#comment-13440511
]
Jason Lowe commented on HADOOP-8723:
Part of the issue is that the sleep job is actually part of
mapreduce-client-jobclient-tests and not mapreduce-examples. In general test
classes will need to be added to the classpath when running tests (not
examples), assuming it's OK for tests of one component to rely on test code
from another component.
But I agree that the sleep job is used heavily enough that we should at least
reduce its dependencies so it can run without classpath changes. Arguably it
should be promoted into mapreduce-examples.
Remove tests and tests-sources jars from classpath
--
Key: HADOOP-8723
URL: https://issues.apache.org/jira/browse/HADOOP-8723
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: HADOOP-8723.patch
Currently {{hadoop-config.sh}} is including any tests and tests-sources jars
in the default classpath, as those jars are shipped in the dist tarball next
to the main jars and the script is globbing everything in that directory.
The tests and tests-sources jars aren't required to run Hadoop, but they can
cause breakage when inadvertently picked up. See HDFS-3831 as an example.
Ideally we should not be adding these jars to the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8225) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/HADOOP-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440514#comment-13440514 ] Hudson commented on HADOOP-8225: Integrated in Hadoop-Hdfs-trunk-Commit #2692 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2692/]) HADOOP-8225. DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376618) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376618 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java DistCp fails when invoked by Oozie -- Key: HADOOP-8225 URL: https://issues.apache.org/jira/browse/HADOOP-8225 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.1, 2.0.0-alpha, 3.0.0 Reporter: Mithun Radhakrishnan Assignee: Daryn Sharp Priority: Blocker Fix For: 0.23.3, 2.1.0-alpha, 3.0.0, 2.2.0-alpha Attachments: HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8225) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/HADOOP-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440517#comment-13440517 ] Hudson commented on HADOOP-8225: Integrated in Hadoop-Common-trunk-Commit #2628 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2628/]) HADOOP-8225. DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376618) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376618 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java DistCp fails when invoked by Oozie -- Key: HADOOP-8225 URL: https://issues.apache.org/jira/browse/HADOOP-8225 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.1, 2.0.0-alpha, 3.0.0 Reporter: Mithun Radhakrishnan Assignee: Daryn Sharp Priority: Blocker Fix For: 0.23.3, 2.1.0-alpha, 3.0.0, 2.2.0-alpha Attachments: HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8723) Remove tests and tests-sources jars from classpath
[
https://issues.apache.org/jira/browse/HADOOP-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440546#comment-13440546
]
Hadoop QA commented on HADOOP-8723:
---
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542157/HADOOP-8723.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-assemblies.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1354//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1354//console
This message is automatically generated.
Remove tests and tests-sources jars from classpath
--
Key: HADOOP-8723
URL: https://issues.apache.org/jira/browse/HADOOP-8723
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: HADOOP-8723.patch
Currently {{hadoop-config.sh}} is including any tests and tests-sources jars
in the default classpath, as those jars are shipped in the dist tarball next
to the main jars and the script is globbing everything in that directory.
The tests and tests-sources jars aren't required to run Hadoop, but they can
cause breakage when inadvertently picked up. See HDFS-3831 as an example.
Ideally we should not be adding these jars to the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8724) Add improved APIs for globbing
Robert Joseph Evans created HADOOP-8724: --- Summary: Add improved APIs for globbing Key: HADOOP-8724 URL: https://issues.apache.org/jira/browse/HADOOP-8724 Project: Hadoop Common Issue Type: Improvement Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans After the discussion on HADOOP-8709 it was decided that we need better APIs for globbing to remove some of the inconsistencies with other APIs. Inorder to maintain backwards compatibility we should deprecate the existing APIs and add in new ones. See HADOOP-8709 for more information about exactly how those APIs should look and behave. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8709) globStatus changed behavior from 0.20/1.x
[ https://issues.apache.org/jira/browse/HADOOP-8709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440571#comment-13440571 ] Robert Joseph Evans commented on HADOOP-8709: - I just filed HADOOP-8724 to add in the new API. I assigned it to myself and I will try and post an initial patch in the next day or so for discussion. I am going to check in the patch to maintain 1.0 compatibility now. globStatus changed behavior from 0.20/1.x - Key: HADOOP-8709 URL: https://issues.apache.org/jira/browse/HADOOP-8709 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.23.0, 2.0.0-alpha Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: HADOOP-8709.patch, HADOOP-8709.patch In 0.20 or 1.x, globStatus will return an empty array if the glob pattern does not match any files. After HADOOP-6201 it throws FileNotFoundException. The javadoc states it will return an empty array. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8709) globStatus changed behavior from 0.20/1.x
[ https://issues.apache.org/jira/browse/HADOOP-8709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HADOOP-8709: Resolution: Fixed Fix Version/s: 2.2.0-alpha 3.0.0 2.1.0-alpha 0.23.3 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Jason, I pulled the patch into trunk, branch-2, branch-2.1.0-alpha and branch-0.23. globStatus changed behavior from 0.20/1.x - Key: HADOOP-8709 URL: https://issues.apache.org/jira/browse/HADOOP-8709 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.23.0, 2.0.0-alpha Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Fix For: 0.23.3, 2.1.0-alpha, 3.0.0, 2.2.0-alpha Attachments: HADOOP-8709.patch, HADOOP-8709.patch In 0.20 or 1.x, globStatus will return an empty array if the glob pattern does not match any files. After HADOOP-6201 it throws FileNotFoundException. The javadoc states it will return an empty array. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8225) DistCp fails when invoked by Oozie
[ https://issues.apache.org/jira/browse/HADOOP-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440594#comment-13440594 ] Hudson commented on HADOOP-8225: Integrated in Hadoop-Mapreduce-trunk-Commit #2656 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2656/]) HADOOP-8225. DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376618) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376618 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java DistCp fails when invoked by Oozie -- Key: HADOOP-8225 URL: https://issues.apache.org/jira/browse/HADOOP-8225 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.1, 2.0.0-alpha, 3.0.0 Reporter: Mithun Radhakrishnan Assignee: Daryn Sharp Priority: Blocker Fix For: 0.23.3, 2.1.0-alpha, 3.0.0, 2.2.0-alpha Attachments: HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch, HADOOP-8225.patch When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as: ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp operation: java.lang.SecurityException: Intercepted System.exit(-999) at org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:904) at org.apache.hadoop.tools.DistCp.main(DistCp.java:357) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8709) globStatus changed behavior from 0.20/1.x
[ https://issues.apache.org/jira/browse/HADOOP-8709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440597#comment-13440597 ] Hudson commented on HADOOP-8709: Integrated in Hadoop-Common-trunk-Commit #2629 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2629/]) HADOOP-8709. globStatus changed behavior from 0.20/1.x (Jason Lowe via bobby) (Revision 1376653) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376653 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java globStatus changed behavior from 0.20/1.x - Key: HADOOP-8709 URL: https://issues.apache.org/jira/browse/HADOOP-8709 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.23.0, 2.0.0-alpha Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Fix For: 0.23.3, 2.1.0-alpha, 3.0.0, 2.2.0-alpha Attachments: HADOOP-8709.patch, HADOOP-8709.patch In 0.20 or 1.x, globStatus will return an empty array if the glob pattern does not match any files. After HADOOP-6201 it throws FileNotFoundException. The javadoc states it will return an empty array. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8709) globStatus changed behavior from 0.20/1.x
[ https://issues.apache.org/jira/browse/HADOOP-8709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440601#comment-13440601 ] Hudson commented on HADOOP-8709: Integrated in Hadoop-Hdfs-trunk-Commit #2693 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2693/]) HADOOP-8709. globStatus changed behavior from 0.20/1.x (Jason Lowe via bobby) (Revision 1376653) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376653 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java globStatus changed behavior from 0.20/1.x - Key: HADOOP-8709 URL: https://issues.apache.org/jira/browse/HADOOP-8709 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.23.0, 2.0.0-alpha Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Fix For: 0.23.3, 2.1.0-alpha, 3.0.0, 2.2.0-alpha Attachments: HADOOP-8709.patch, HADOOP-8709.patch In 0.20 or 1.x, globStatus will return an empty array if the glob pattern does not match any files. After HADOOP-6201 it throws FileNotFoundException. The javadoc states it will return an empty array. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8712) Change default hadoop.security.group.mapping
[ https://issues.apache.org/jira/browse/HADOOP-8712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440650#comment-13440650 ] Robert Parker commented on HADOOP-8712: --- HDFS-3837 and HDFS-3850 addresses the findbugs issues Change default hadoop.security.group.mapping Key: HADOOP-8712 URL: https://issues.apache.org/jira/browse/HADOOP-8712 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.2.0-alpha Reporter: Robert Parker Assignee: Robert Parker Priority: Minor Attachments: HADOOP-8712-v1.patch Change the hadoop.security.group.mapping in core-site to JniBasedUnixGroupsNetgroupMappingWithFallback -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8709) globStatus changed behavior from 0.20/1.x
[ https://issues.apache.org/jira/browse/HADOOP-8709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440653#comment-13440653 ] Sanjay Radia commented on HADOOP-8709: -- +1 on the patch. globStatus changed behavior from 0.20/1.x - Key: HADOOP-8709 URL: https://issues.apache.org/jira/browse/HADOOP-8709 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.23.0, 2.0.0-alpha Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Fix For: 0.23.3, 2.1.0-alpha, 3.0.0, 2.2.0-alpha Attachments: HADOOP-8709.patch, HADOOP-8709.patch In 0.20 or 1.x, globStatus will return an empty array if the glob pattern does not match any files. After HADOOP-6201 it throws FileNotFoundException. The javadoc states it will return an empty array. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8709) globStatus changed behavior from 0.20/1.x
[ https://issues.apache.org/jira/browse/HADOOP-8709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440656#comment-13440656 ] Hudson commented on HADOOP-8709: Integrated in Hadoop-Mapreduce-trunk-Commit #2657 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2657/]) HADOOP-8709. globStatus changed behavior from 0.20/1.x (Jason Lowe via bobby) (Revision 1376653) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376653 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java globStatus changed behavior from 0.20/1.x - Key: HADOOP-8709 URL: https://issues.apache.org/jira/browse/HADOOP-8709 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.23.0, 2.0.0-alpha Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Fix For: 0.23.3, 2.1.0-alpha, 3.0.0, 2.2.0-alpha Attachments: HADOOP-8709.patch, HADOOP-8709.patch In 0.20 or 1.x, globStatus will return an empty array if the glob pattern does not match any files. After HADOOP-6201 it throws FileNotFoundException. The javadoc states it will return an empty array. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8705) Add JSR 107 Caching support
[ https://issues.apache.org/jira/browse/HADOOP-8705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440723#comment-13440723 ] Ahmed Radwan commented on HADOOP-8705: -- Thanks Dhruv, This is interesting. I think the work on having pluggable MapOutputBuffer and shuffle can highly facilitate such effort. Add JSR 107 Caching support Key: HADOOP-8705 URL: https://issues.apache.org/jira/browse/HADOOP-8705 Project: Hadoop Common Issue Type: Improvement Reporter: Dhruv Kumar Having a cache on mappers and reducers could be very useful for some use cases, including but not limited to: 1. Iterative Map Reduce Programs: Some machine learning algorithms frequently need access to invariant data (see Mahout) over each iteration of MapReduce until convergence. A cache on such nodes could allow easy access to the hotset of data without going all the way to the distributed cache. 2. Storing of intermediate map and reduce outputs in memory to reduce shuffling time. This optimization has been discussed at length in Haloop (http://www.ics.uci.edu/~yingyib/papers/HaLoop_camera_ready.pdf). There are some other scenarios as well where having a cache could come in handy. It will be nice to have some sort of pluggable support for JSR 107 compliant caches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7745) I switched variable names in HADOOP-7509
[ https://issues.apache.org/jira/browse/HADOOP-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-7745: --- Target Version/s: (was: 0.24.0) Fix Version/s: 1.1.0 Was not properly log messaged, so I can't find when this was merged to branch-1, but apparently it was after the branch-1.0 branchpoint. Adding 1.1.0 to the FixedIn list. I switched variable names in HADOOP-7509 Key: HADOOP-7745 URL: https://issues.apache.org/jira/browse/HADOOP-7745 Project: Hadoop Common Issue Type: Bug Reporter: Ravi Prakash Assignee: Ravi Prakash Fix For: 1.1.0, 0.23.0, 0.24.0 Attachments: patchToCorrect7509.branch-0.20.patch, patchToCorrect7509.patch As Aaron pointed out on https://issues.apache.org/jira/browse/HADOOP-7509?focusedCommentId=13126725page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13126725 I stupidly swapped CommonConfigurationKeys.HADOOP_SECURITY_AUTHENTICATION with CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-7942) enabling clover coverage reports fails hadoop unit test compilation
[
https://issues.apache.org/jira/browse/HADOOP-7942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley resolved HADOOP-7942.
Resolution: Fixed
Seems like no one has observed a need for this in other branches in six months.
Resolving as part of 1.1.0 release cleanup.
enabling clover coverage reports fails hadoop unit test compilation
---
Key: HADOOP-7942
URL: https://issues.apache.org/jira/browse/HADOOP-7942
Project: Hadoop Common
Issue Type: Test
Affects Versions: 1.1.0
Environment:
https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-1-Code-Coverage/13/console
Reporter: Giridharan Kesavan
Assignee: Jitendra Nath Pandey
Fix For: 1.1.0
Attachments: HADOOP-7942.branch-1.patch
enabling clover reports fails compiling the following junit tests.
link to the console output of jerkins :
https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-1-Code-Coverage/13/console
{noformat}
[javac]
/tmp/clover50695626838999169.tmp/org/apache/hadoop/security/TestUserGroupInformation.java:224:
cannot find symbol
..
[javac]
/tmp/clover50695626838999169.tmp/org/apache/hadoop/security/TestUserGroupInformation.java:225:
cannot find symbol
..
[javac]
/tmp/clover50695626838999169.tmp/org/apache/hadoop/security/TestJobCredentials.java:67:
cannot find symbol
[javac] symbol : class T
..
[javac]
/tmp/clover50695626838999169.tmp/org/apache/hadoop/security/TestJobCredentials.java:68:
cannot find symbol
[javac] symbol : class T
.
[javac]
/tmp/clover50695626838999169.tmp/org/apache/hadoop/fs/TestFileSystem.java:653:
cannot find symbol
[javac] symbol : class T
.
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 5 errors
[javac] 63 warnings
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8314) HttpServer#hasAdminAccess should return false if authorization is enabled but user is not authenticated
[ https://issues.apache.org/jira/browse/HADOOP-8314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-8314: --- Fix Version/s: 1.1.0 HttpServer#hasAdminAccess should return false if authorization is enabled but user is not authenticated --- Key: HADOOP-8314 URL: https://issues.apache.org/jira/browse/HADOOP-8314 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 1.1.0, 2.0.0-alpha, 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 1.1.0, 2.0.0-alpha Attachments: HADOOP-8314_branch-1.patch, HADOOP-8314.patch, HADOOP-8314.patch If the user is not authenticated (request.getRemoteUser() returns NULL) or there is not authentication filter configured (thus returning also NULL), hasAdminAccess should return false. Note that a filter could allow anonymous access, thus the first case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8031) Configuration class fails to find embedded .jar resources; should use URL.openStream()
[
https://issues.apache.org/jira/browse/HADOOP-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440761#comment-13440761
]
Alejandro Abdelnur commented on HADOOP-8031:
+1
Configuration class fails to find embedded .jar resources; should use
URL.openStream()
--
Key: HADOOP-8031
URL: https://issues.apache.org/jira/browse/HADOOP-8031
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.0.0-alpha
Reporter: Elias Ross
Assignee: Elias Ross
Attachments: 0001-fix-HADOOP-7982-class-loader.patch,
HADOOP-8031.patch, hadoop-8031.txt
While running a hadoop client within RHQ (monitoring software) using its
classloader, I see this:
2012-02-07 09:15:25,313 INFO [ResourceContainer.invoker.daemon-2]
(org.apache.hadoop.conf.Configuration)- parsing
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
2012-02-07 09:15:25,318 ERROR [InventoryManager.discovery-1]
(rhq.core.pc.inventory.InventoryManager)- Failed to start component for
Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20,
name=NameNode, parent=vg61l01ad-hadoop002.apple.com] from synchronized merge.
org.rhq.core.clientapi.agent.PluginContainerException: Failed to start
component for resource Resource[id=16290, type=NameNode,
key=NameNode:/usr/lib/hadoop-0.20, name=NameNode,
parent=vg61l01ad-hadoop002.apple.com].
Caused by: java.lang.RuntimeException: core-site.xml not found
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1308)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1228)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1169)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)
This is because the URL
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
cannot be found by DocumentBuilder (doesn't understand it). (Note: the logs
are for an old version of Configuration class, but the new version has the
same code.)
The solution is to obtain the resource stream directly from the URL object
itself.
That is to say:
{code}
URL url = getResource((String)name);
-if (url != null) {
- if (!quiet) {
-LOG.info(parsing + url);
- }
- doc = builder.parse(url.toString());
-}
+doc = builder.parse(url.openStream());
{code}
Note: I have a full patch pending approval at Apple for this change,
including some cleanup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8031) Configuration class fails to find embedded .jar resources; should use URL.openStream()
[
https://issues.apache.org/jira/browse/HADOOP-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HADOOP-8031:
---
Resolution: Fixed
Fix Version/s: 2.2.0-alpha
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Elias. Committed to trunk and branch-2.
Configuration class fails to find embedded .jar resources; should use
URL.openStream()
--
Key: HADOOP-8031
URL: https://issues.apache.org/jira/browse/HADOOP-8031
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.0.0-alpha
Reporter: Elias Ross
Assignee: Elias Ross
Fix For: 2.2.0-alpha
Attachments: 0001-fix-HADOOP-7982-class-loader.patch,
HADOOP-8031.patch, hadoop-8031.txt
While running a hadoop client within RHQ (monitoring software) using its
classloader, I see this:
2012-02-07 09:15:25,313 INFO [ResourceContainer.invoker.daemon-2]
(org.apache.hadoop.conf.Configuration)- parsing
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
2012-02-07 09:15:25,318 ERROR [InventoryManager.discovery-1]
(rhq.core.pc.inventory.InventoryManager)- Failed to start component for
Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20,
name=NameNode, parent=vg61l01ad-hadoop002.apple.com] from synchronized merge.
org.rhq.core.clientapi.agent.PluginContainerException: Failed to start
component for resource Resource[id=16290, type=NameNode,
key=NameNode:/usr/lib/hadoop-0.20, name=NameNode,
parent=vg61l01ad-hadoop002.apple.com].
Caused by: java.lang.RuntimeException: core-site.xml not found
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1308)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1228)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1169)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)
This is because the URL
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
cannot be found by DocumentBuilder (doesn't understand it). (Note: the logs
are for an old version of Configuration class, but the new version has the
same code.)
The solution is to obtain the resource stream directly from the URL object
itself.
That is to say:
{code}
URL url = getResource((String)name);
-if (url != null) {
- if (!quiet) {
-LOG.info(parsing + url);
- }
- doc = builder.parse(url.toString());
-}
+doc = builder.parse(url.openStream());
{code}
Note: I have a full patch pending approval at Apple for this change,
including some cleanup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8031) Configuration class fails to find embedded .jar resources; should use URL.openStream()
[
https://issues.apache.org/jira/browse/HADOOP-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440794#comment-13440794
]
Hudson commented on HADOOP-8031:
Integrated in Hadoop-Common-trunk-Commit #2632 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2632/])
HADOOP-8031. Configuration class fails to find embedded .jar resources;
should use URL.openStream() (genman via tucu) (Revision 1376772)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376772
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
Configuration class fails to find embedded .jar resources; should use
URL.openStream()
--
Key: HADOOP-8031
URL: https://issues.apache.org/jira/browse/HADOOP-8031
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.0.0-alpha
Reporter: Elias Ross
Assignee: Elias Ross
Fix For: 2.2.0-alpha
Attachments: 0001-fix-HADOOP-7982-class-loader.patch,
HADOOP-8031.patch, hadoop-8031.txt
While running a hadoop client within RHQ (monitoring software) using its
classloader, I see this:
2012-02-07 09:15:25,313 INFO [ResourceContainer.invoker.daemon-2]
(org.apache.hadoop.conf.Configuration)- parsing
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
2012-02-07 09:15:25,318 ERROR [InventoryManager.discovery-1]
(rhq.core.pc.inventory.InventoryManager)- Failed to start component for
Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20,
name=NameNode, parent=vg61l01ad-hadoop002.apple.com] from synchronized merge.
org.rhq.core.clientapi.agent.PluginContainerException: Failed to start
component for resource Resource[id=16290, type=NameNode,
key=NameNode:/usr/lib/hadoop-0.20, name=NameNode,
parent=vg61l01ad-hadoop002.apple.com].
Caused by: java.lang.RuntimeException: core-site.xml not found
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1308)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1228)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1169)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)
This is because the URL
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
cannot be found by DocumentBuilder (doesn't understand it). (Note: the logs
are for an old version of Configuration class, but the new version has the
same code.)
The solution is to obtain the resource stream directly from the URL object
itself.
That is to say:
{code}
URL url = getResource((String)name);
-if (url != null) {
- if (!quiet) {
-LOG.info(parsing + url);
- }
- doc = builder.parse(url.toString());
-}
+doc = builder.parse(url.openStream());
{code}
Note: I have a full patch pending approval at Apple for this change,
including some cleanup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-4572) INode and its sub-classes should be package private
[ https://issues.apache.org/jira/browse/HADOOP-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440793#comment-13440793 ] Hudson commented on HADOOP-4572: Integrated in Hadoop-Common-trunk-Commit #2632 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2632/]) HADOOP-4572. Can not access user logs - Jetty is not configured by default to serve aliases/symlinks (ahmed via tucu) (Revision 1376753) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376753 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml INode and its sub-classes should be package private --- Key: HADOOP-4572 URL: https://issues.apache.org/jira/browse/HADOOP-4572 Project: Hadoop Common Issue Type: Improvement Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.20.0 Attachments: 4572_20081103.patch, svn_move.sh INode, INodeFile, INodeDirectory and INodeFileUnderConstruction should be changed from public to package private. As a consequence, some test classes like CreateEditsLog have to be moved to org.apache.hadoop.hdfs.server.namenode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-4572) INode and its sub-classes should be package private
[ https://issues.apache.org/jira/browse/HADOOP-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440802#comment-13440802 ] Hudson commented on HADOOP-4572: Integrated in Hadoop-Hdfs-trunk-Commit #2696 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2696/]) HADOOP-4572. Can not access user logs - Jetty is not configured by default to serve aliases/symlinks (ahmed via tucu) (Revision 1376753) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376753 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml INode and its sub-classes should be package private --- Key: HADOOP-4572 URL: https://issues.apache.org/jira/browse/HADOOP-4572 Project: Hadoop Common Issue Type: Improvement Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.20.0 Attachments: 4572_20081103.patch, svn_move.sh INode, INodeFile, INodeDirectory and INodeFileUnderConstruction should be changed from public to package private. As a consequence, some test classes like CreateEditsLog have to be moved to org.apache.hadoop.hdfs.server.namenode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8031) Configuration class fails to find embedded .jar resources; should use URL.openStream()
[
https://issues.apache.org/jira/browse/HADOOP-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440803#comment-13440803
]
Hudson commented on HADOOP-8031:
Integrated in Hadoop-Hdfs-trunk-Commit #2696 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2696/])
HADOOP-8031. Configuration class fails to find embedded .jar resources;
should use URL.openStream() (genman via tucu) (Revision 1376772)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376772
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
Configuration class fails to find embedded .jar resources; should use
URL.openStream()
--
Key: HADOOP-8031
URL: https://issues.apache.org/jira/browse/HADOOP-8031
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.0.0-alpha
Reporter: Elias Ross
Assignee: Elias Ross
Fix For: 2.2.0-alpha
Attachments: 0001-fix-HADOOP-7982-class-loader.patch,
HADOOP-8031.patch, hadoop-8031.txt
While running a hadoop client within RHQ (monitoring software) using its
classloader, I see this:
2012-02-07 09:15:25,313 INFO [ResourceContainer.invoker.daemon-2]
(org.apache.hadoop.conf.Configuration)- parsing
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
2012-02-07 09:15:25,318 ERROR [InventoryManager.discovery-1]
(rhq.core.pc.inventory.InventoryManager)- Failed to start component for
Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20,
name=NameNode, parent=vg61l01ad-hadoop002.apple.com] from synchronized merge.
org.rhq.core.clientapi.agent.PluginContainerException: Failed to start
component for resource Resource[id=16290, type=NameNode,
key=NameNode:/usr/lib/hadoop-0.20, name=NameNode,
parent=vg61l01ad-hadoop002.apple.com].
Caused by: java.lang.RuntimeException: core-site.xml not found
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1308)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1228)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1169)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)
This is because the URL
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
cannot be found by DocumentBuilder (doesn't understand it). (Note: the logs
are for an old version of Configuration class, but the new version has the
same code.)
The solution is to obtain the resource stream directly from the URL object
itself.
That is to say:
{code}
URL url = getResource((String)name);
-if (url != null) {
- if (!quiet) {
-LOG.info(parsing + url);
- }
- doc = builder.parse(url.toString());
-}
+doc = builder.parse(url.openStream());
{code}
Note: I have a full patch pending approval at Apple for this change,
including some cleanup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7461) Jackson Dependency Not Declared in Hadoop POM
[ https://issues.apache.org/jira/browse/HADOOP-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-7461: --- Fix Version/s: (was: 1.1.0) This was indeed fixed in 1.0.0, so no need to note it as fixed in 1.1.0. Jackson Dependency Not Declared in Hadoop POM - Key: HADOOP-7461 URL: https://issues.apache.org/jira/browse/HADOOP-7461 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 0.20.205.0 Reporter: Ron Bodkin Assignee: Giridharan Kesavan Fix For: 1.0.0 Attachments: HADOOP-7461-1.patch, HADOOP-7461.patch (COMMENT: This bug still affects 0.20.205.0, four months after the bug was filed. This causes total failure, and the fix is trivial for whoever manages the POM -- just add the missing dependency! --ben) This issue was identified and the fix workaround was documented at https://issues.cloudera.org/browse/DISTRO-44 The issue affects use of Hadoop 0.20.203.0 from the Maven central repo. I built a job using that maven repo and ran it, resulting in this failure: Exception in thread main java.lang.NoClassDefFoundError: org/codehaus/jackson/map/JsonMappingException at thinkbig.hadoop.inputformat.TestXmlInputFormat.run(TestXmlInputFormat.java:18) at thinkbig.hadoop.inputformat.TestXmlInputFormat.main(TestXmlInputFormat.java:23) Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.map.JsonMappingException -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7645) HTTP auth tests requiring Kerberos infrastructure are not disabled on branch-0.20-security
[
https://issues.apache.org/jira/browse/HADOOP-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated HADOOP-7645:
---
Fix Version/s: (was: 1.1.0)
Fixed in 20.205, which was a lineal predecessor of 1.0. No need to mark it
fixed in 1.1.0.
HTTP auth tests requiring Kerberos infrastructure are not disabled on
branch-0.20-security
--
Key: HADOOP-7645
URL: https://issues.apache.org/jira/browse/HADOOP-7645
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 0.20.205.0
Reporter: Aaron T. Myers
Assignee: Jitendra Nath Pandey
Priority: Blocker
Fix For: 0.20.205.0
Attachments: hadoop-7645-022.patch, HADOOP-7645-20s.1.patch,
HADOOP-7645-20s.2.patch
The back-port of HADOOP-7119 to branch-0.20-security included tests which
require Kerberos infrastructure in order to run. In trunk and 0.23, these are
disabled unless one enables the {{testKerberos}} maven profile. In
branch-0.20-security, these tests are always run regardless, and so fail most
of the time.
See this Jenkins build for an example:
https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-0.20-security/26/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7664) o.a.h.conf.Configuration complains of overriding final parameter even if the value with which its attempting to override is the same.
[ https://issues.apache.org/jira/browse/HADOOP-7664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-7664: --- Target Version/s: 1.0.0, 0.24.0 (was: 1.1.0, 0.24.0) Fix Version/s: (was: 1.1.0) correct Fixed Versions. o.a.h.conf.Configuration complains of overriding final parameter even if the value with which its attempting to override is the same. -- Key: HADOOP-7664 URL: https://issues.apache.org/jira/browse/HADOOP-7664 Project: Hadoop Common Issue Type: Improvement Components: conf Affects Versions: 0.20.205.0, 0.23.0 Environment: commit a2f64ee8d9312fe24780ec53b15af439a315796d Reporter: Ravi Prakash Assignee: Ravi Prakash Priority: Minor Fix For: 1.0.0, 0.23.0 Attachments: HADOOP-7664.branch-0.20-security.patch, HADOOP-7664.patch, HADOOP-7664.patch, HADOOP-7664.patch Original Estimate: 1m Time Spent: 1m Remaining Estimate: 0h o.a.h.conf.Configuration complains of overriding final parameter even if the value with which its attempting to override is the same. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7674) TestKerberosName fails in 20 branch.
[ https://issues.apache.org/jira/browse/HADOOP-7674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-7674: --- Fix Version/s: (was: 1.1.0) 20.205 was a lineal predecessor of 1.0. TestKerberosName fails in 20 branch. Key: HADOOP-7674 URL: https://issues.apache.org/jira/browse/HADOOP-7674 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.20.205.0 Attachments: HADOOP-7674-20s.1.patch, hadoop-7674-22.patch TestKerberosName fails in 20 branch. In fact this test has got duplicated in 20, with a little change to the rules. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7987) Support setting the run-as user in unsecure mode
[ https://issues.apache.org/jira/browse/HADOOP-7987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-7987: --- Fix Version/s: (was: 1.1.0) 1.0.1 was a lineal predecessor of 1.1.0. Support setting the run-as user in unsecure mode Key: HADOOP-7987 URL: https://issues.apache.org/jira/browse/HADOOP-7987 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 1.0.0, 0.23.0, 0.24.0 Reporter: Devaraj Das Assignee: Jitendra Nath Pandey Fix For: 1.0.1, 0.23.1, 0.24.0 Attachments: HADOOP-7987.branch-1.patch, HADOOP-7987.trunk.patch Some applications need to be able to perform actions (such as launch MR jobs) from map or reduce tasks. In earlier unsecure versions of hadoop (20.x), it was possible to do this by setting user.name in the configuration. But in 20.205 and 1.0, when running in unsecure mode, this does not work. (In secure mode, you can do this using the kerberos credentials). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8293) The native library's Makefile.am doesn't include JNI path
[
https://issues.apache.org/jira/browse/HADOOP-8293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated HADOOP-8293:
---
Fix Version/s: (was: 1.1.0)
1.0.3 was an ancestor of 1.1.0.
The native library's Makefile.am doesn't include JNI path
-
Key: HADOOP-8293
URL: https://issues.apache.org/jira/browse/HADOOP-8293
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Fix For: 1.0.3
Attachments: h-8293.patch
When compiling on centos 6, I get the following error when compiling the
native library:
{code}
[exec] /usr/bin/ld: cannot find -ljvm
{code}
The problem is simply that the Makefile.am libhadoop_la_LDFLAGS doesn't
include AM_LDFLAGS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8251) SecurityUtil.fetchServiceTicket broken after HADOOP-6941
[ https://issues.apache.org/jira/browse/HADOOP-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-8251: --- Target Version/s: 2.0.0-alpha, 1.0.3 (was: 1.1.0, 2.0.0-alpha) Fix Version/s: (was: 1.1.0) 1.0.3 was an ancestor of 1.1.0. SecurityUtil.fetchServiceTicket broken after HADOOP-6941 Key: HADOOP-8251 URL: https://issues.apache.org/jira/browse/HADOOP-8251 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 1.1.0, 2.0.0-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Fix For: 1.0.3, 2.0.0-alpha Attachments: hadoop-8251-b1.txt, hadoop-8251.txt HADOOP-6941 replaced direct references to some classes with reflective access so as to support other JDKs. Unfortunately there was a mistake in the name of the Krb5Util class, which broke fetchServiceTicket. This manifests itself as the inability to run checkpoints or other krb5-SSL HTTP-based transfers: java.lang.ClassNotFoundException: sun.security.jgss.krb5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8338) Can't renew or cancel HDFS delegation tokens over secure RPC
[ https://issues.apache.org/jira/browse/HADOOP-8338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-8338: --- Fix Version/s: (was: 1.1.0) 1.0.3 was an ancestor of 1.1.0. Can't renew or cancel HDFS delegation tokens over secure RPC Key: HADOOP-8338 URL: https://issues.apache.org/jira/browse/HADOOP-8338 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 1.0.3 Attachments: hadoop-8338.patch The fetchdt tool is failing for secure deployments when given --renew or --cancel on tokens fetched using RPC. (The tokens fetched over HTTP can be renewed and canceled fine.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HADOOP-8338) Can't renew or cancel HDFS delegation tokens over secure RPC
[ https://issues.apache.org/jira/browse/HADOOP-8338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440815#comment-13440815 ] Matt Foley edited comment on HADOOP-8338 at 8/24/12 11:16 AM: -- 1.0.3 was a lineal predecessor of 1.1.0. was (Author: mattf): 1.0.3 was an ancestor of 1.1.0. Can't renew or cancel HDFS delegation tokens over secure RPC Key: HADOOP-8338 URL: https://issues.apache.org/jira/browse/HADOOP-8338 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 1.0.3 Attachments: hadoop-8338.patch The fetchdt tool is failing for secure deployments when given --renew or --cancel on tokens fetched using RPC. (The tokens fetched over HTTP can be renewed and canceled fine.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HADOOP-8251) SecurityUtil.fetchServiceTicket broken after HADOOP-6941
[ https://issues.apache.org/jira/browse/HADOOP-8251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440813#comment-13440813 ] Matt Foley edited comment on HADOOP-8251 at 8/24/12 11:16 AM: -- 1.0.3 was a lineal predecessor of 1.1.0. was (Author: mattf): 1.0.3 was an ancestor of 1.1.0. SecurityUtil.fetchServiceTicket broken after HADOOP-6941 Key: HADOOP-8251 URL: https://issues.apache.org/jira/browse/HADOOP-8251 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 1.1.0, 2.0.0-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Fix For: 1.0.3, 2.0.0-alpha Attachments: hadoop-8251-b1.txt, hadoop-8251.txt HADOOP-6941 replaced direct references to some classes with reflective access so as to support other JDKs. Unfortunately there was a mistake in the name of the Krb5Util class, which broke fetchServiceTicket. This manifests itself as the inability to run checkpoints or other krb5-SSL HTTP-based transfers: java.lang.ClassNotFoundException: sun.security.jgss.krb5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HADOOP-8293) The native library's Makefile.am doesn't include JNI path
[
https://issues.apache.org/jira/browse/HADOOP-8293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440814#comment-13440814
]
Matt Foley edited comment on HADOOP-8293 at 8/24/12 11:16 AM:
--
1.0.3 was a lineal predecessor of 1.1.0.
was (Author: mattf):
1.0.3 was an ancestor of 1.1.0.
The native library's Makefile.am doesn't include JNI path
-
Key: HADOOP-8293
URL: https://issues.apache.org/jira/browse/HADOOP-8293
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Fix For: 1.0.3
Attachments: h-8293.patch
When compiling on centos 6, I get the following error when compiling the
native library:
{code}
[exec] /usr/bin/ld: cannot find -ljvm
{code}
The problem is simply that the Makefile.am libhadoop_la_LDFLAGS doesn't
include AM_LDFLAGS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8352) We should always generate a new configure script for the c++ code
[ https://issues.apache.org/jira/browse/HADOOP-8352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-8352: --- Fix Version/s: (was: 1.1.0) 1.0.3 was a lineal predecessor of 1.1.0. We should always generate a new configure script for the c++ code - Key: HADOOP-8352 URL: https://issues.apache.org/jira/browse/HADOOP-8352 Project: Hadoop Common Issue Type: Improvement Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 1.0.3 Attachments: gen-c++.lst, git-ignore.patch, hadoop-8352.patch If you are compiling c++, you should always generate a configure script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-4572) INode and its sub-classes should be package private
[ https://issues.apache.org/jira/browse/HADOOP-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440848#comment-13440848 ] Hudson commented on HADOOP-4572: Integrated in Hadoop-Mapreduce-trunk-Commit #2660 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2660/]) HADOOP-4572. Can not access user logs - Jetty is not configured by default to serve aliases/symlinks (ahmed via tucu) (Revision 1376753) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376753 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml INode and its sub-classes should be package private --- Key: HADOOP-4572 URL: https://issues.apache.org/jira/browse/HADOOP-4572 Project: Hadoop Common Issue Type: Improvement Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.20.0 Attachments: 4572_20081103.patch, svn_move.sh INode, INodeFile, INodeDirectory and INodeFileUnderConstruction should be changed from public to package private. As a consequence, some test classes like CreateEditsLog have to be moved to org.apache.hadoop.hdfs.server.namenode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8031) Configuration class fails to find embedded .jar resources; should use URL.openStream()
[
https://issues.apache.org/jira/browse/HADOOP-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440849#comment-13440849
]
Hudson commented on HADOOP-8031:
Integrated in Hadoop-Mapreduce-trunk-Commit #2660 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2660/])
HADOOP-8031. Configuration class fails to find embedded .jar resources;
should use URL.openStream() (genman via tucu) (Revision 1376772)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1376772
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
Configuration class fails to find embedded .jar resources; should use
URL.openStream()
--
Key: HADOOP-8031
URL: https://issues.apache.org/jira/browse/HADOOP-8031
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 2.0.0-alpha
Reporter: Elias Ross
Assignee: Elias Ross
Fix For: 2.2.0-alpha
Attachments: 0001-fix-HADOOP-7982-class-loader.patch,
HADOOP-8031.patch, hadoop-8031.txt
While running a hadoop client within RHQ (monitoring software) using its
classloader, I see this:
2012-02-07 09:15:25,313 INFO [ResourceContainer.invoker.daemon-2]
(org.apache.hadoop.conf.Configuration)- parsing
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
2012-02-07 09:15:25,318 ERROR [InventoryManager.discovery-1]
(rhq.core.pc.inventory.InventoryManager)- Failed to start component for
Resource[id=16290, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20,
name=NameNode, parent=vg61l01ad-hadoop002.apple.com] from synchronized merge.
org.rhq.core.clientapi.agent.PluginContainerException: Failed to start
component for resource Resource[id=16290, type=NameNode,
key=NameNode:/usr/lib/hadoop-0.20, name=NameNode,
parent=vg61l01ad-hadoop002.apple.com].
Caused by: java.lang.RuntimeException: core-site.xml not found
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1308)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1228)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1169)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)
This is because the URL
jar:file:/usr/local/rhq-agent/data/tmp/rhq-hadoop-plugin-4.3.0-SNAPSHOT.jar6856622641102893436.classloader/hadoop-core-0.20.2+737+1.jar7204287718482036191.tmp!/core-default.xml
cannot be found by DocumentBuilder (doesn't understand it). (Note: the logs
are for an old version of Configuration class, but the new version has the
same code.)
The solution is to obtain the resource stream directly from the URL object
itself.
That is to say:
{code}
URL url = getResource((String)name);
-if (url != null) {
- if (!quiet) {
-LOG.info(parsing + url);
- }
- doc = builder.parse(url.toString());
-}
+doc = builder.parse(url.openStream());
{code}
Note: I have a full patch pending approval at Apple for this change,
including some cleanup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8725) MR is broken when security is off
Daryn Sharp created HADOOP-8725: --- Summary: MR is broken when security is off Key: HADOOP-8725 URL: https://issues.apache.org/jira/browse/HADOOP-8725 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.3, 2.1.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker HADOOP-8225 broke MR when security is off. MR was changed to stop re-reading the credentials that UGI had already read, and to stop putting those tokens back into the UGI where they already were. UGI only reads a credentials file when security is enabled, but MR uses tokens (ie. job token) even when security is disabled... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8725) MR is broken when security is off
[ https://issues.apache.org/jira/browse/HADOOP-8725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-8725: Attachment: HADOOP-8725.patch I'm not sure how to easily write a test since there's not a supported way to set envs in java. Well, that's non-extremely-dirty-and-hacky by using reflection to bypass the immutable map. MR is broken when security is off - Key: HADOOP-8725 URL: https://issues.apache.org/jira/browse/HADOOP-8725 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.3, 2.1.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-8725.patch HADOOP-8225 broke MR when security is off. MR was changed to stop re-reading the credentials that UGI had already read, and to stop putting those tokens back into the UGI where they already were. UGI only reads a credentials file when security is enabled, but MR uses tokens (ie. job token) even when security is disabled... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8725) MR is broken when security is off
[ https://issues.apache.org/jira/browse/HADOOP-8725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-8725: Status: Patch Available (was: Open) MR is broken when security is off - Key: HADOOP-8725 URL: https://issues.apache.org/jira/browse/HADOOP-8725 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.3, 2.1.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-8725.patch HADOOP-8225 broke MR when security is off. MR was changed to stop re-reading the credentials that UGI had already read, and to stop putting those tokens back into the UGI where they already were. UGI only reads a credentials file when security is enabled, but MR uses tokens (ie. job token) even when security is disabled... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8568) DNS#reverseDns fails on IPv6 addresses
[
https://issues.apache.org/jira/browse/HADOOP-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440880#comment-13440880
]
Andy Isaacson commented on HADOOP-8568:
---
{code}
+int rawintaddr = rawaddr[i] 0 ? rawaddr[i] + 256 : rawaddr[i];
{code}
It's a bit simpler and faster (and has the same results) to say
{code}
int rawintaddr = rawaddr[i] 0xff;
{code}
but I would have thought that there would be a standard idiom for treating a
byte[] as a list of unsigned 8 bit values. Maybe not?
bq. Revised patch, uses StringBuffer to build the reverseIP String
Doesn't look like an updated upload happened, the current attachment has string
concatenation in a loop.
DNS#reverseDns fails on IPv6 addresses
--
Key: HADOOP-8568
URL: https://issues.apache.org/jira/browse/HADOOP-8568
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Tony Kew
Labels: newbie
Attachments: HADOOP-8568.patch
DNS#reverseDns assumes hostIp is a v4 address (4 parts separated by dots),
blows up if given a v6 address:
{noformat}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 3
at org.apache.hadoop.net.DNS.reverseDns(DNS.java:79)
at org.apache.hadoop.net.DNS.getHosts(DNS.java:237)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:340)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:358)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:337)
at org.apache.hadoop.hbase.master.HMaster.init(HMaster.java:235)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1649)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8725) MR is broken when security is off
[ https://issues.apache.org/jira/browse/HADOOP-8725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440914#comment-13440914 ] Robert Joseph Evans commented on HADOOP-8725: - I have manually verified that the changes fix the problem. The change is small and looks good to me +1 assuming Jenkins comes back OK. MR is broken when security is off - Key: HADOOP-8725 URL: https://issues.apache.org/jira/browse/HADOOP-8725 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.3, 2.1.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-8725.patch HADOOP-8225 broke MR when security is off. MR was changed to stop re-reading the credentials that UGI had already read, and to stop putting those tokens back into the UGI where they already were. UGI only reads a credentials file when security is enabled, but MR uses tokens (ie. job token) even when security is disabled... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8725) MR is broken when security is off
[ https://issues.apache.org/jira/browse/HADOOP-8725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440916#comment-13440916 ] Hadoop QA commented on HADOOP-8725: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542227/HADOOP-8725.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1355//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1355//console This message is automatically generated. MR is broken when security is off - Key: HADOOP-8725 URL: https://issues.apache.org/jira/browse/HADOOP-8725 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.3, 2.1.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker Attachments: HADOOP-8725.patch HADOOP-8225 broke MR when security is off. MR was changed to stop re-reading the credentials that UGI had already read, and to stop putting those tokens back into the UGI where they already were. UGI only reads a credentials file when security is enabled, but MR uses tokens (ie. job token) even when security is disabled... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8725) MR is broken when security is off
[
https://issues.apache.org/jira/browse/HADOOP-8725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440917#comment-13440917
]
Alejandro Abdelnur commented on HADOOP-8725:
I've just run TestMRJobs with the patch and I'm still seen errors
{code}
---
Running org.apache.hadoop.mapreduce.v2.TestMRJobs
Tests run: 4, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 119.838 sec
FAILURE!
Results :
Failed tests: testSleepJob(org.apache.hadoop.mapreduce.v2.TestMRJobs)
testRandomWriter(org.apache.hadoop.mapreduce.v2.TestMRJobs)
testDistributedCache(org.apache.hadoop.mapreduce.v2.TestMRJobs)
{code}
Unless they are unrelated, the issue is still there.
MR is broken when security is off
-
Key: HADOOP-8725
URL: https://issues.apache.org/jira/browse/HADOOP-8725
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 0.23.3, 2.1.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Blocker
Attachments: HADOOP-8725.patch
HADOOP-8225 broke MR when security is off. MR was changed to stop re-reading
the credentials that UGI had already read, and to stop putting those tokens
back into the UGI where they already were. UGI only reads a credentials file
when security is enabled, but MR uses tokens (ie. job token) even when
security is disabled...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HADOOP-8725) MR is broken when security is off
[
https://issues.apache.org/jira/browse/HADOOP-8725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440922#comment-13440922
]
Alejandro Abdelnur edited comment on HADOOP-8725 at 8/24/12 2:49 PM:
-
mmh, jenkings come back while I was writing my prev comment. But I believe
jenkins is not testing MR, can somebody verify running {{mvn clean test
-Dtest=TestMRJobs}}?
was (Author: tucu00):
mmh, jenkings come back while I was writing my prev comment. But I believe
jenkins is not testing MR, can somebody verify running {mvn clean test
-Dtest=TestMRJobs}?
MR is broken when security is off
-
Key: HADOOP-8725
URL: https://issues.apache.org/jira/browse/HADOOP-8725
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 0.23.3, 2.1.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Blocker
Attachments: HADOOP-8725.patch
HADOOP-8225 broke MR when security is off. MR was changed to stop re-reading
the credentials that UGI had already read, and to stop putting those tokens
back into the UGI where they already were. UGI only reads a credentials file
when security is enabled, but MR uses tokens (ie. job token) even when
security is disabled...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8725) MR is broken when security is off
[
https://issues.apache.org/jira/browse/HADOOP-8725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13440922#comment-13440922
]
Alejandro Abdelnur commented on HADOOP-8725:
mmh, jenkings come back while I was writing my prev comment. But I believe
jenkins is not testing MR, can somebody verify running {mvn clean test
-Dtest=TestMRJobs}?
MR is broken when security is off
-
Key: HADOOP-8725
URL: https://issues.apache.org/jira/browse/HADOOP-8725
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 0.23.3, 2.1.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Blocker
Attachments: HADOOP-8725.patch
HADOOP-8225 broke MR when security is off. MR was changed to stop re-reading
the credentials that UGI had already read, and to stop putting those tokens
back into the UGI where they already were. UGI only reads a credentials file
when security is enabled, but MR uses tokens (ie. job token) even when
security is disabled...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
