[jira] [Comment Edited] (HADOOP-6700) Hadoop commands guide should include examples
[ https://issues.apache.org/jira/browse/HADOOP-6700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079615#comment-14079615 ] Allen Wittenauer edited comment on HADOOP-6700 at 7/30/14 5:43 PM: --- There are some examples for some stuff now, but not nearly as many as there should be. :D was (Author: aw): There are some examples for some stuff now, but not nearly as many as their should be. :D Hadoop commands guide should include examples - Key: HADOOP-6700 URL: https://issues.apache.org/jira/browse/HADOOP-6700 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Amareshwari Sriramadasu Labels: newbie Currently, The Hadoop command guide (http://hadoop.apache.org/common/docs/r0.20.0/commands_manual.html) just lists all the available command line options, with a description. It should include examples for each command for more clarity. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-7266) Deprecate metrics v1
[ https://issues.apache.org/jira/browse/HADOOP-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-7266. -- Resolution: Fixed Fixed. I think. Deprecate metrics v1 Key: HADOOP-7266 URL: https://issues.apache.org/jira/browse/HADOOP-7266 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 0.21.0 Reporter: Luke Lu Assignee: Luke Lu -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6728) Overhaul metrics framework
[ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6728. -- Resolution: Fixed Closing as fixed! Overhaul metrics framework -- Key: HADOOP-6728 URL: https://issues.apache.org/jira/browse/HADOOP-6728 Project: Hadoop Common Issue Type: Improvement Affects Versions: 0.20.2, 0.21.0 Reporter: Luke Lu Assignee: Luke Lu Attachments: hadoop-6728-y20.104.patch, metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to: * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508). * Refresh metrics plugin config without server restart. ** Including filtering of metrics per plugin. * Support metrics schema for plugins. The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues. [The current design wiki|http://wiki.apache.org/hadoop/HADOOP-6728-MetricsV2]. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6729) serializer.JavaSerialization should be added to io.serializations by default
[ https://issues.apache.org/jira/browse/HADOOP-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6729. -- Resolution: Won't Fix Closing as won't fix based upon Tom's comments. So it's all his fault. serializer.JavaSerialization should be added to io.serializations by default Key: HADOOP-6729 URL: https://issues.apache.org/jira/browse/HADOOP-6729 Project: Hadoop Common Issue Type: Improvement Components: conf Affects Versions: 0.20.2 Reporter: Ted Yu org.apache.hadoop.io.serializer.JavaSerialization isn't included in io.serializations by default. When a class which implements the Serializable interface is used, user would see the following without serializer.JavaSerialization: java.lang.NullPointerException at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:759) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:487) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:575) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.Child.main(Child.java:170) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-8984) S3 File Permissions
[ https://issues.apache.org/jira/browse/HADOOP-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-8984. -- Resolution: Incomplete Likely stale. S3 File Permissions --- Key: HADOOP-8984 URL: https://issues.apache.org/jira/browse/HADOOP-8984 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.2-alpha Environment: Hadoop cluster using 3 small Amazon EC2 machines and the S3FileSystem. Hadoop compiled from latest trunc: 0.22.0-SNAPSHOT core-site: fs.default.name=s3://my-s3-bucket fs.s3.awsAccessKeyId=[key id omitted] fs.s3.awsSecretAccessKey=[secret key omitted] hadoop.tmp.dir=/mnt/hadoop.tmp.dir hdfs-site: empty mapred-site: mapred.job.tracker=[domU-XX-XX-XX-XX-XX-XX.compute-1.internal:9001] mapred.map.tasks=6 mapred.reduce.tasks=6 Reporter: Danny Leshem Priority: Critical Till lately I've been using 0.20.2 and everything was ok. Now I'm using the latest trunc 0.22.0-SNAPSHOT and getting the following thrown: Exception in thread main java.io.IOException: The ownership/permissions on the staging directory s3://my-s3-bucket/mnt/hadoop.tmp.dir/mapred/staging/root/.staging is not as expected. It is owned by and permissions are rwxrwxrwx. The directory must be owned by the submitter root or by root and permissions must be rwx-- at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:107) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:312) at org.apache.hadoop.mapreduce.Job.submit(Job.java:961) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:977) at com.mycompany.MyJob.runJob(MyJob.java:153) at com.mycompany.MyJob.run(MyJob.java:177) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at com.mycompany.MyOtherJob.runJob(MyOtherJob.java:62) at com.mycompany.MyOtherJob.run(MyOtherJob.java:112) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at com.mycompany.MyOtherJob.main(MyOtherJob.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:187) (The it is owned by ... and permissions is not a mistake, seems like the empty string is printed there) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-6731) Octal umask with less than 3 digits is not handled
[ https://issues.apache.org/jira/browse/HADOOP-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-6731: - Labels: newbie (was: ) Octal umask with less than 3 digits is not handled -- Key: HADOOP-6731 URL: https://issues.apache.org/jira/browse/HADOOP-6731 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.21.0, 0.22.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Labels: newbie Attachments: HADOOP-6731.patch Umasks with less than three digits should be accepted as valid umask. Umask with single digit, say 7, should be handled as 007. Similarly umask with two digits, say 77, should be handled as 077. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6733) Create a test for FileSystem API compatibility between releases
[ https://issues.apache.org/jira/browse/HADOOP-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6733. -- Resolution: Won't Fix Hi Big Top. Closing. Create a test for FileSystem API compatibility between releases --- Key: HADOOP-6733 URL: https://issues.apache.org/jira/browse/HADOOP-6733 Project: Hadoop Common Issue Type: Sub-task Components: fs Reporter: Tom White We should have an automated test for checking that programs written against an old version of the FileSystem API still run with a newer version. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6732) Improve FsShell's heap consumption by switching to listStatus that returns an iterator
[ https://issues.apache.org/jira/browse/HADOOP-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6732. -- Resolution: Incomplete I know FsShell has undergone several revisions since this JIRA was filed. Closing as stale. If it is still an issue, let's open a new one. Hooray! Improve FsShell's heap consumption by switching to listStatus that returns an iterator -- Key: HADOOP-6732 URL: https://issues.apache.org/jira/browse/HADOOP-6732 Project: Hadoop Common Issue Type: Improvement Reporter: Hairong Kuang Assignee: Daryn Sharp When listing a large directory from the command line using the default heap configuration, FsShell often runs out of memory. This is because all stats of the entries under the directory need to be in memory before printing them. The new API listStatus that returns an iterator of FileStatus, which implemented in HDFS-1091, no longer requires that all entries are fetched first. Thus switching to this new API will greatly improve the use of heap space. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6744) Pull out FS_DEFAULT_NAME definition from FileSystem and move it to CommonConfigurationKeys
[ https://issues.apache.org/jira/browse/HADOOP-6744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6744. -- Resolution: Won't Fix fs.default.name was deprecated. Pull out FS_DEFAULT_NAME definition from FileSystem and move it to CommonConfigurationKeys --- Key: HADOOP-6744 URL: https://issues.apache.org/jira/browse/HADOOP-6744 Project: Hadoop Common Issue Type: Improvement Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6753) MetricsServlet does not show shuffleInput metrics
[ https://issues.apache.org/jira/browse/HADOOP-6753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6753. -- Resolution: Fixed MetricsServlet does not show shuffleInput metrics - Key: HADOOP-6753 URL: https://issues.apache.org/jira/browse/HADOOP-6753 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 0.20.1 Environment: Linux/Mac Reporter: Alex Kozlov Original Estimate: 1h Remaining Estimate: 1h Thanks to Philip and HADOOP-5469.patch. It's very useful. I just found out that it does not show shuffleInput tasktracker metrics. Should be really straightforward to add. Alex K -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6751) hadoop daemonlog does not work from command line with security enabled
[ https://issues.apache.org/jira/browse/HADOOP-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6751. -- Resolution: Fixed hadoop daemonlog does not work from command line with security enabled -- Key: HADOOP-6751 URL: https://issues.apache.org/jira/browse/HADOOP-6751 Project: Hadoop Common Issue Type: Bug Reporter: Jitendra Nath Pandey daemonlog command line is not working with security enabled. We need to support both browser interface and command line with security enabled for daemonlog. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6759) Contrib Streaming.jar for hadoop-0.20.2
[ https://issues.apache.org/jira/browse/HADOOP-6759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6759. -- Resolution: Cannot Reproduce Contrib Streaming.jar for hadoop-0.20.2 Key: HADOOP-6759 URL: https://issues.apache.org/jira/browse/HADOOP-6759 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 0.20.2 Reporter: Luis Ramos I downloaded release-0.20.2/ from SVN hadoop/common/tags/. I was able to build successfully and use hadoop, but I noticed there is no hadoop-0.20.2-streaming.jar in build/contrib/streaming/ Is there a way to build this separately? I looked everywhere. The packaged download has it fine. Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6765) Duplicate commons-net in published POM
[ https://issues.apache.org/jira/browse/HADOOP-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6765. -- Resolution: Fixed Duplicate commons-net in published POM -- Key: HADOOP-6765 URL: https://issues.apache.org/jira/browse/HADOOP-6765 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 0.20.2 Reporter: Scott Coplin The POM file that was published to the maven central repository for 0.20.2 contains two dependency entries for commons-net. http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core/0.20.2/hadoop-core-0.20.2.pom I believe Maven flags this as a warning and moves on, but it causes other maven-based tools such as M2Eclipse to choke. It would be nice to have this fixed and a corrected POM republished. Looks like the same problem exists on trunk as well in what I presume is the source template for the POM. (http://svn.apache.org/viewvc/hadoop/common/trunk/ivy/hadoop-core-template.xml?view=markup) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6767) Patch for running Hadoop on Windows without Cygwin
[ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6767. -- Resolution: Not a Problem Some company whose headquarters is currently in Redmond gave us patches too. Those patches have been committed making these obsolete. Patch for running Hadoop on Windows without Cygwin -- Key: HADOOP-6767 URL: https://issues.apache.org/jira/browse/HADOOP-6767 Project: Hadoop Common Issue Type: Improvement Components: build, conf, scripts Affects Versions: 0.22.0 Environment: Windows XP, 2003, 7, 2008 Reporter: Volodymyr Orlov Labels: blockdecompressorstream Attachments: HADOOP-6767.patch, Hadoop-0.20.2-patched.zip, hadoop-6767-r1.tar.gz Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6775) Update Hadoop Common Sites
[ https://issues.apache.org/jira/browse/HADOOP-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6775. -- Resolution: Fixed This has been done (e.g. http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/InterfaceClassification.html ). Update Hadoop Common Sites --- Key: HADOOP-6775 URL: https://issues.apache.org/jira/browse/HADOOP-6775 Project: Hadoop Common Issue Type: Improvement Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: site Attachments: api_classification.pdf, siteCLassification.patch, siteCLassification2.patch, siteCLassification3.patch Add documentation on our interface classification scheme to thew common site. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6795) FsShell 'hadoop fs -text' does not work with other file systems
[ https://issues.apache.org/jira/browse/HADOOP-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6795. -- Resolution: Fixed FsShell 'hadoop fs -text' does not work with other file systems - Key: HADOOP-6795 URL: https://issues.apache.org/jira/browse/HADOOP-6795 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.20.2 Reporter: Shunsuke Mikami Assignee: Brian Bloniarz Priority: Minor Labels: patch Attachments: hadoop-6795-branch-1.patch, hadoop-6795.patch FsShell 'hadoop fs -text' can only work with file system which set by fs.default.name. I use Gfarm file system from Hadoop. https://gfarm.svn.sourceforge.net/svnroot/gfarm/gfarm_hadoop/trunk/ If i set fs.default.name to hdfs, the error Wrong FS occurred when i submit 'hadoop fs -text' to file on gfarm file system. $ hadoop fs -text gfarmfs:///home/mikami/random/part-0 text: Wrong FS: gfarmfs://null/home/mikami/random/part-0, expected: hdfs://hostname:9000 if i set fs.default.name to gfarmfs:///, i can get correct result. this command's result shouldn't depend on fs.default.name. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-6795) FsShell 'hadoop fs -text' does not work with other file systems
[ https://issues.apache.org/jira/browse/HADOOP-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-6795: - Target Version/s: (was: 1.3.0) FsShell 'hadoop fs -text' does not work with other file systems - Key: HADOOP-6795 URL: https://issues.apache.org/jira/browse/HADOOP-6795 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.20.2 Reporter: Shunsuke Mikami Assignee: Brian Bloniarz Priority: Minor Labels: patch Attachments: hadoop-6795-branch-1.patch, hadoop-6795.patch FsShell 'hadoop fs -text' can only work with file system which set by fs.default.name. I use Gfarm file system from Hadoop. https://gfarm.svn.sourceforge.net/svnroot/gfarm/gfarm_hadoop/trunk/ If i set fs.default.name to hdfs, the error Wrong FS occurred when i submit 'hadoop fs -text' to file on gfarm file system. $ hadoop fs -text gfarmfs:///home/mikami/random/part-0 text: Wrong FS: gfarmfs://null/home/mikami/random/part-0, expected: hdfs://hostname:9000 if i set fs.default.name to gfarmfs:///, i can get correct result. this command's result shouldn't depend on fs.default.name. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-6783) Rename core-* configuration files to common-*
[
https://issues.apache.org/jira/browse/HADOOP-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079850#comment-14079850
]
Allen Wittenauer commented on HADOOP-6783:
--
Meanwhile, four years later...
I suggest we do [~enis] suggestion in the 3.x trunk. It's a great time to
clean that up.
Rename core-* configuration files to common-*
-
Key: HADOOP-6783
URL: https://issues.apache.org/jira/browse/HADOOP-6783
Project: Hadoop Common
Issue Type: Bug
Components: conf
Reporter: Chris Douglas
To reflect the name of the project, the {{core-site.xml}} and
{{core-default.xml}} configuration files should be renamed to {{common-}} and
the references in documentation (including javadoc) should be updated.
Compatibility should be maintained, with a warning.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-6797) DNS#getHosts() fallback leads to mix of network-interface addresse in case reverse lookup fails
[
https://issues.apache.org/jira/browse/HADOOP-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-6797.
--
Resolution: Incomplete
I'm going to close this as stale at this point.
DNS#getHosts() fallback leads to mix of network-interface addresse in case
reverse lookup fails
---
Key: HADOOP-6797
URL: https://issues.apache.org/jira/browse/HADOOP-6797
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.20.2
Reporter: Johannes Zillmann
org.apache.hadoop.net.DNS#getHosts():
{noformat}
public static String[] getHosts(String strInterface, String nameserver)
throws UnknownHostException {
String[] ips = getIPs(strInterface);
VectorString hosts = new VectorString();
for (int ctr = 0; ctr ips.length; ctr++)
try {
hosts.add(reverseDns(InetAddress.getByName(ips[ctr]),
nameserver));
} catch (Exception e) {
}
if (hosts.size() == 0)
return new String[] { InetAddress.getLocalHost().getCanonicalHostName()
};
else
return hosts.toArray(new String[] {});
}
{noformat}
I have the situation where i choosing eth1 as network interface
(fs.datanode.dns.interface and mapred.tasktracker.dns.interface) which is the
internal interface.
The reverse lookup fails for eth1 so the fallback:
{noformat}
if (hosts.size() == 0)
return new String[] { InetAddress.getLocalHost().getCanonicalHostName()
};
{noformat}
comes to action. The dns of eth0 is returned which is the external interface.
This leads in my case to a combination of internal ip and external dns as the
default ip/host.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-6804) allow FileSystem.copyFromLocalFile() to execute under specified username
[ https://issues.apache.org/jira/browse/HADOOP-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6804. -- Resolution: Not a Problem allow FileSystem.copyFromLocalFile() to execute under specified username Key: HADOOP-6804 URL: https://issues.apache.org/jira/browse/HADOOP-6804 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 0.20.2 Reporter: Ted Yu Priority: Minor When the user calling FileSystem.copyFromLocalFile() doesn't have permission to write to certain hdfs path: Thread [main] (Suspended (exception AccessControlException)) DFSClient.mkdirs(String, FsPermission) line: 905 DistributedFileSystem.mkdirs(Path, FsPermission) line: 262 DistributedFileSystem(FileSystem).mkdirs(Path) line: 1162 FileUtil.copy(FileSystem, Path, FileSystem, Path, boolean, boolean, Configuration) line: 194 DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, boolean, Path, Path) line: 1231 DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, Path, Path) line: 1207 DistributedFileSystem(FileSystem).copyFromLocalFile(Path, Path) line: 1179 GridM2mInstallation.copyInputFiles(FlowConfigurations$FlowConf) line: 380 Passwordless ssh has been setup for current user, tyu, on localhost and user hadoop on hdfs FileSystem.copyFromLocalFile() should be able to execute using a different username than effective user on localhost so that AccessControlException can be avoided. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6806) While running HADOOP Exception in thread main java.lang.NullPointerException occurs
[ https://issues.apache.org/jira/browse/HADOOP-6806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6806. -- Resolution: Cannot Reproduce While running HADOOP Exception in thread main java.lang.NullPointerException occurs Key: HADOOP-6806 URL: https://issues.apache.org/jira/browse/HADOOP-6806 Project: Hadoop Common Issue Type: Task Environment: CentOS 5.0 Reporter: GIRISH N IYER hi, while installin hadoop ... i can do the following step bin/hadoop namenode -format...it got success After when i do bin/start-all.shthe following error occurs starting namenode, logging to /hadoop/hadoop/bin/../logs/hadoop-hadoop-namenode-localhost.localdomain.out localhost: ssh: localhost: Name or service not known hadoop@master's password: master: starting secondarynamenode, logging to /hadoop/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-localhost.localdomain.out master: Exception in thread main java.lang.NullPointerException master: at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134) master: at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156) master: at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160) master: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:131) master: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.init(SecondaryNameNode.java:115) master: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:469) starting jobtracker, logging to /hadoop/hadoop/bin/../logs/hadoop-hadoop-jobtracker-localhost.localdomain.out localhost: ssh: localhost: Name or service not known Pls help me to solve this Also i did not configure nano conf/core-site.xml nano conf/hdfs-site.xml nano conf/mapred-site.xml -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (HADOOP-7266) Deprecate metrics v1
[ https://issues.apache.org/jira/browse/HADOOP-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer reopened HADOOP-7266: -- Nope, looks like metricsv1 is still there. Deprecate metrics v1 Key: HADOOP-7266 URL: https://issues.apache.org/jira/browse/HADOOP-7266 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.0.0 Reporter: Luke Lu Assignee: Luke Lu -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-7266) Deprecate metrics v1
[ https://issues.apache.org/jira/browse/HADOOP-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-7266: - Issue Type: Improvement (was: Sub-task) Parent: (was: HADOOP-6728) Deprecate metrics v1 Key: HADOOP-7266 URL: https://issues.apache.org/jira/browse/HADOOP-7266 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Luke Lu Assignee: Luke Lu -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-7266) Deprecate metrics v1
[ https://issues.apache.org/jira/browse/HADOOP-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-7266: - Affects Version/s: (was: 0.21.0) 3.0.0 Deprecate metrics v1 Key: HADOOP-7266 URL: https://issues.apache.org/jira/browse/HADOOP-7266 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.0.0 Reporter: Luke Lu Assignee: Luke Lu -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6808) Document steps to enable {File|Ganglia}Context for kerberos metrics
[
https://issues.apache.org/jira/browse/HADOOP-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-6808.
--
Resolution: Fixed
Document steps to enable {File|Ganglia}Context for kerberos metrics
---
Key: HADOOP-6808
URL: https://issues.apache.org/jira/browse/HADOOP-6808
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 0.20.2, 0.22.0
Reporter: Erik Steffl
Priority: Trivial
Attachments: HADOOP-6808-0.20.1xx.patch
There was apatch adding kerberos metrics which included the following line in
conf/hadoop-metrics.properties:
ugi.class=org.apache.hadoop.metrics.spi.NullContext
However to maintain consistency with dfs,jvm and mapred metrics and for
documentation purpose we should include comments in the file to include steps
to enable FileContext and GangliaContext as well. The following lines have to
be included in conf/hadoop-metrics.properties:
# Configuration of the ugi context for file
#ugi.class=org.apache.hadoop.metrics.file.FileContext
#ugi.period=10
#ugi.fileName=/tmp/ugimetrics.log
# Configuration of the ugi context for ganglia
# ugi.class=org.apache.hadoop.metrics.ganglia.GangliaContext
# ugi.period=10
# ugi.servers=localhost:8649
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-6816) enhance FsShell.dus() to include sum of totalSize
[ https://issues.apache.org/jira/browse/HADOOP-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-6816: - Labels: newbie (was: ) enhance FsShell.dus() to include sum of totalSize - Key: HADOOP-6816 URL: https://issues.apache.org/jira/browse/HADOOP-6816 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 0.20.2 Reporter: Ted Yu Labels: newbie FsShell.dus() prints out totalSize for each file found. It would be desirable to print sum of totalSize's at the end. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-6824) Improve NetUtils:normalizeHostName
[ https://issues.apache.org/jira/browse/HADOOP-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-6824: - Labels: newbie (was: ) Improve NetUtils:normalizeHostName -- Key: HADOOP-6824 URL: https://issues.apache.org/jira/browse/HADOOP-6824 Project: Hadoop Common Issue Type: Improvement Reporter: Jakob Homan Labels: newbie HADOOP-6682 revealed a bug in NetUtils:normalizeHostName, which got fixed, but it may be worth replacing the current best-effort code with a more rigorous approach, as Hong suggested. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6820) RunJar fails executing thousands JARs within single JVM with error Too many open files
[ https://issues.apache.org/jira/browse/HADOOP-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6820. -- Resolution: Fixed This is a system tuning and/or JVM bug. Closing as won't fix. RunJar fails executing thousands JARs within single JVM with error Too many open files Key: HADOOP-6820 URL: https://issues.apache.org/jira/browse/HADOOP-6820 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 0.20.2 Environment: OS:Linux, Linux-user limited by maximum number of open file descriptors (for example: ulimit -n shows 1024) Reporter: Alexander Bondar Priority: Minor Attachments: HADOOP-6820.patch According to Sun JVM (up to 7) bug http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4167874 - The JarFile objects created by sun.net.www.protocol.jar.JarFileFactory never get garbage collected, even if the classloader that loaded them goes away. So, if linux-user has limitation on maximum number of open file descriptors (for example: ulimit -n shows 1024) and performs RunJar.main(...) over thousands of JARs that include other nested JARs (also loaded by ClassLoader) within single JVM, RunJar.main(...) throws following exception: java.lang.RuntimeException: java.io.FileNotFoundException: /some-file.txt (Too many open files) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6829) Deprecated scripts shouldn't be used by Herriot
[
https://issues.apache.org/jira/browse/HADOOP-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-6829.
--
Resolution: Fixed
Deprecated scripts shouldn't be used by Herriot
---
Key: HADOOP-6829
URL: https://issues.apache.org/jira/browse/HADOOP-6829
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 0.21.0
Reporter: Konstantin Boudnik
{{hadoop-daemon.sh}} has been phased out in 0.21 and its usage for HDFS and
MR was replaced by {{hdfs}} and {{mapred}} scripts.
Herriot shouldn't be using the deprecated script.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-6830) Javadoc warnings in trunk.
[
https://issues.apache.org/jira/browse/HADOOP-6830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-6830.
--
Resolution: Fixed
Stale.
Javadoc warnings in trunk.
--
Key: HADOOP-6830
URL: https://issues.apache.org/jira/browse/HADOOP-6830
Project: Hadoop Common
Issue Type: Bug
Reporter: Amareshwari Sriramadasu
Trunk has following javadoc warnings :
{noformat}
[exec] [javadoc]
/grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:31:
warning: sun.security.krb5.Config is Sun proprietary API and may be removed
in a future release
[exec] [javadoc] import sun.security.krb5.Config;
[exec] [javadoc] ^
[exec] [javadoc]
/grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:32:
warning: sun.security.krb5.KrbException is Sun proprietary API and may be
removed in a future release
[exec] [javadoc] import sun.security.krb5.KrbException;
[exec] [javadoc] ^
[exec] [javadoc]
/grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:81:
warning: sun.security.krb5.Config is Sun proprietary API and may be removed
in a future release
[exec] [javadoc] private static Config kerbConf;
[exec] [javadoc] ^
[exec] [javadoc]
/grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:33:
warning: sun.security.jgss.krb5.Krb5Util is Sun proprietary API and may be
removed in a future release
[exec] [javadoc] import sun.security.jgss.krb5.Krb5Util;
[exec] [javadoc] ^
[exec] [javadoc]
/grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:34:
warning: sun.security.krb5.Credentials is Sun proprietary API and may be
removed in a future release
[exec] [javadoc] import sun.security.krb5.Credentials;
[exec] [javadoc] ^
[exec] [javadoc]
/grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:35:
warning: sun.security.krb5.PrincipalName is Sun proprietary API and may be
removed in a future release
[exec] [javadoc] import sun.security.krb5.PrincipalName;
[exec] [javadoc]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Moved] (HADOOP-10906) getConetentSummary() for HarFileSystem throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/HADOOP-10906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer moved MAPREDUCE-1877 to HADOOP-10906:
--
Component/s: (was: harchive)
Affects Version/s: (was: 0.22.0)
Key: HADOOP-10906 (was: MAPREDUCE-1877)
Project: Hadoop Common (was: Hadoop Map/Reduce)
getConetentSummary() for HarFileSystem throws IllegalArgumentException
--
Key: HADOOP-10906
URL: https://issues.apache.org/jira/browse/HADOOP-10906
Project: Hadoop Common
Issue Type: Bug
Reporter: Paul Yang
As HarFileSystem does not implement getContentSummary(), the implementation
from FilterFileSystem is inherited by default. However,
FilterFileSystem.getContentSummary() does not work for the HarFileSystem
because the method attempts to use HarFileSystem's underlying FS to call
getContentSummary(). In the case where the the underlying filesystem is HDFS,
an exception similar to the following is thrown:
{code}
java.lang.IllegalArgumentException: Wrong FS:
har://hdfs-example.com:9000/tmp/data.har, expected: hdfs://example.com:9000
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:352)
at
org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:99)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:155)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getContentSummary(DistributedFileSystem.java:232)
at
org.apache.hadoop.fs.FilterFileSystem.getContentSummary(FilterFileSystem.java:287)
at
org.apache.hadoop.fs.FilterFileSystem.getContentSummary(FilterFileSystem.java:287)
{code}
One solution is to implement HarFileSystem.getContentSummary() using code
similar to FileSystem.getContentSummary().
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-6838) Investigate Eclipse API Tools for enforcing or reporting on API compatibility
[ https://issues.apache.org/jira/browse/HADOOP-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6838. -- Resolution: Incomplete Better handled by Big Top. Closing. Investigate Eclipse API Tools for enforcing or reporting on API compatibility -- Key: HADOOP-6838 URL: https://issues.apache.org/jira/browse/HADOOP-6838 Project: Hadoop Common Issue Type: Improvement Components: build, documentation Reporter: Tom White Some links: * http://eclipsesource.com/blogs/2010/06/14/api_tools_top_eclipse_helios_feature_8/ * http://wiki.eclipse.org/PDE/API_Tools -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6837) Support for LZMA compression
[ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6837. -- Resolution: Won't Fix Support for LZMA compression Key: HADOOP-6837 URL: https://issues.apache.org/jira/browse/HADOOP-6837 Project: Hadoop Common Issue Type: Improvement Components: io Reporter: Nicholas Carlini Assignee: Nicholas Carlini Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-4-20100811.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10907) Single Node Setup uses JDK 1.6 and bin/*-all.sh scripts
[ https://issues.apache.org/jira/browse/HADOOP-10907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10907: -- Labels: newbie (was: ) Single Node Setup uses JDK 1.6 and bin/*-all.sh scripts --- Key: HADOOP-10907 URL: https://issues.apache.org/jira/browse/HADOOP-10907 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Allen Wittenauer Labels: newbie JDK 1.6 is deprecated the *-all.sh scripts are now in sbin. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10907) Single Node Setup uses JDK 1.6 and bin/*-all.sh scripts
Allen Wittenauer created HADOOP-10907: - Summary: Single Node Setup uses JDK 1.6 and bin/*-all.sh scripts Key: HADOOP-10907 URL: https://issues.apache.org/jira/browse/HADOOP-10907 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Allen Wittenauer JDK 1.6 is deprecated the *-all.sh scripts are now in sbin. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10908) Cluster Node Setup needs updating post-HADOOP-9902
[ https://issues.apache.org/jira/browse/HADOOP-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10908: -- Summary: Cluster Node Setup needs updating post-HADOOP-9902 (was: Cluster Node Setup needs updating post-9902) Cluster Node Setup needs updating post-HADOOP-9902 -- Key: HADOOP-10908 URL: https://issues.apache.org/jira/browse/HADOOP-10908 Project: Hadoop Common Issue Type: Bug Reporter: Allen Wittenauer A lot of the instructions in the cluster node setup are not good practices post-9902. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10908) Cluster Node Setup needs updating post-9902
Allen Wittenauer created HADOOP-10908: - Summary: Cluster Node Setup needs updating post-9902 Key: HADOOP-10908 URL: https://issues.apache.org/jira/browse/HADOOP-10908 Project: Hadoop Common Issue Type: Bug Reporter: Allen Wittenauer A lot of the instructions in the cluster node setup are not good practices post-9902. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10907) Single Node Setup still thinks it is hadoop 1.x
[ https://issues.apache.org/jira/browse/HADOOP-10907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10907: -- Description: # JDK 1.6 is deprecated # the *-all.sh scripts are now in sbin. # location of hadoop-*-examples.jar is no longer there # hadoop jar should be replaced with yarn jar was: JDK 1.6 is deprecated the *-all.sh scripts are now in sbin. Summary: Single Node Setup still thinks it is hadoop 1.x (was: Single Node Setup uses JDK 1.6 and bin/*-all.sh scripts) Single Node Setup still thinks it is hadoop 1.x --- Key: HADOOP-10907 URL: https://issues.apache.org/jira/browse/HADOOP-10907 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Allen Wittenauer Labels: newbie # JDK 1.6 is deprecated # the *-all.sh scripts are now in sbin. # location of hadoop-*-examples.jar is no longer there # hadoop jar should be replaced with yarn jar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6841) In Quick Start, mention requirement that HADOOP_HOME needs to be set
[ https://issues.apache.org/jira/browse/HADOOP-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6841. -- Resolution: Fixed In Quick Start, mention requirement that HADOOP_HOME needs to be set Key: HADOOP-6841 URL: https://issues.apache.org/jira/browse/HADOOP-6841 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: 0.20.2 Reporter: Steve McCarthy Priority: Minor Ref: http://hadoop.apache.org/common/docs/r0.20.0/quickstart.html#Local Ref: http://wiki.apache.org/hadoop/QuickStart As a person new to Hadoop, I was walking through the Quick Start pages referenced above. I got stuck on the first example and had to go digging through the scripts to figure out why it wasn't working for me. It turns out that I didn't have the HADOOP_HOME environment variable set. It would have saved a lot of time if the instructions mentioned setting this variable. Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HADOOP-10908) Cluster Node Setup needs updating post-HADOOP-9902
[ https://issues.apache.org/jira/browse/HADOOP-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer reassigned HADOOP-10908: - Assignee: Allen Wittenauer Cluster Node Setup needs updating post-HADOOP-9902 -- Key: HADOOP-10908 URL: https://issues.apache.org/jira/browse/HADOOP-10908 Project: Hadoop Common Issue Type: Bug Reporter: Allen Wittenauer Assignee: Allen Wittenauer A lot of the instructions in the cluster node setup are not good practices post-9902. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (HADOOP-10909) Add more documents about command daemonlog
[ https://issues.apache.org/jira/browse/HADOOP-10909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer moved HDFS-1278 to HADOOP-10909: - Component/s: (was: documentation) documentation Assignee: (was: Jeff Zhang) Key: HADOOP-10909 (was: HDFS-1278) Project: Hadoop Common (was: Hadoop HDFS) Add more documents about command daemonlog Key: HADOOP-10909 URL: https://issues.apache.org/jira/browse/HADOOP-10909 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Jeff Zhang In the current document, it does not explain the argument name means. People without java knowledge would think it as process name, such as NameNode or DataNode, but actually it is class name. So I suggest to add more document about the daemonlog to explain more about the arguments. PS: I only find the document on official site (Programming Guide -- Commands Guide), but did not found the document in trunk. Anybody know where's the document of Commands Guide ? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6849) Have LocalDirAllocator.AllocatorPerContext.getLocalPathForWrite fail more meaningfully
[
https://issues.apache.org/jira/browse/HADOOP-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-6849.
--
Resolution: Fixed
A lot of this has been changed. Closing this as stale.
Have LocalDirAllocator.AllocatorPerContext.getLocalPathForWrite fail more
meaningfully
--
Key: HADOOP-6849
URL: https://issues.apache.org/jira/browse/HADOOP-6849
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 0.20.2
Reporter: Steve Loughran
Priority: Minor
A stack trace makes it way to me, of a reduce failing
{code}
Caused by: org\.apache\.hadoop\.util\.DiskChecker$DiskErrorException: Could
not find any valid local directory for
file:/mnt/data/dfs/data/mapred/local/taskTracker/jobcache/job_201007011427_0001/attempt_201007011427_0001_r_00_1/output/map_96\.out
at
org\.apache\.hadoop\.fs\.LocalDirAllocator$AllocatorPerContext\.getLocalPathForWrite(LocalDirAllocator\.java:343)
at
org\.apache\.hadoop\.fs\.LocalDirAllocator\.getLocalPathForWrite(LocalDirAllocator\.java:124)
at
org\.apache\.hadoop\.mapred\.ReduceTask$ReduceCopier$LocalFSMerger\.run(ReduceTask\.java:2434)
{code}
We're probably running out of HDD space, if not its configuration problems.
Either way, some more hints in the exception would be handy.
# Include the size of the output file looked for if known
# Include the list of dirs examined and their reason for rejection (not found
or if not enough room, available space).
This would make it easier to diagnose problems after the event, with nothing
but emailed logs for diagnostics.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-6848) FsShell have resource leak
[ https://issues.apache.org/jira/browse/HADOOP-6848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6848. -- Resolution: Incomplete FsShell got reworked. Closing as Stale. FsShell have resource leak -- Key: HADOOP-6848 URL: https://issues.apache.org/jira/browse/HADOOP-6848 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.20-append Reporter: Vladimir Klimontovich When FsShell exectutes -text command it's using TextRecordInputStream. This class doesn't close inbuf and outbuf (with underlying socket in case of HDFS fs) opened in constructor. It's ok in classic command per JVM case when -text command is being used from command-line. But when FsShell is being used in same JVM several times (for example via http://martiansoftware.com/nailgun/index.html project) it cause socket leak. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-6746) hadoop-daemon.sh does not append to log files
[ https://issues.apache.org/jira/browse/HADOOP-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-6746: - Resolution: Duplicate Target Version/s: 2.3.0, 3.0.0 (was: 3.0.0, 2.3.0) Status: Resolved (was: Patch Available) I'll include this fix into the next patch for HADOOP-9902. Closing as a dupe. hadoop-daemon.sh does not append to log files - Key: HADOOP-6746 URL: https://issues.apache.org/jira/browse/HADOOP-6746 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.2.0 Reporter: Travis Crawford Assignee: Akira AJISAKA Labels: script Attachments: HADOOP-6746.2.patch, HADOOP-6746.patch Looking in hadoop-daemon.sh we see the start command redirects logs to the log file by creating a new file. It does not append to the log. nohup nice -n $HADOOP_NICENESS $HADOOP_HOME/bin/hadoop --config $HADOOP_CONF_DIR $command $@ $log 21 /dev/null I believe output should be appended to the log file so logrotate copytruncate works. Currently a large log file cannot be truncated without restarting the running service, which causes issues in production environments. The improved line would be exactly the same, except use to open the log file in append mode. nohup nice -n $HADOOP_NICENESS $HADOOP_HOME/bin/hadoop --config $HADOOP_CONF_DIR $command $@ $log 21 /dev/null -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6101) CDPATH environment variable causes bin scripts to fail.
[ https://issues.apache.org/jira/browse/HADOOP-6101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6101. -- Resolution: Duplicate Next version of HADOOP-9902 will include this fix. Closing as a dupe. CDPATH environment variable causes bin scripts to fail. --- Key: HADOOP-6101 URL: https://issues.apache.org/jira/browse/HADOOP-6101 Project: Hadoop Common Issue Type: Bug Components: scripts Affects Versions: 0.19.1 Reporter: Phil Hagelberg Priority: Minor Most of the scripts in bin/* assume that cd produces no output. But when using bash (and some other shells) cd will output the destination directory if the CDPATH environment variable is set. CDPATH is very useful, and it's unfortunate to have to unset it to use Hadoop. The offending line (in start-all.sh, though most of the scripts exhibit the problem) is: bin=`cd $bin; pwd` Adding this to the top of each affected script will fix the problem: unset CDPATH -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6851) Fix '$bin' path duplication in setup scripts
[ https://issues.apache.org/jira/browse/HADOOP-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6851. -- Resolution: Duplicate next patch version of HADOOP-9902 will include this fix. Closing as a dupe. Fix '$bin' path duplication in setup scripts Key: HADOOP-6851 URL: https://issues.apache.org/jira/browse/HADOOP-6851 Project: Hadoop Common Issue Type: Bug Components: scripts Reporter: Nicolas Spiegelberg Priority: Trivial Attachments: HADOOP-6851-0.22.patch I have my bash environment setup to echo absolute pathnames when a relative one is specified in 'cd'. This caused problems with all the Hadoop bash scripts because the script accidentally sets the $bin variable twice in this setup. (e.g. would set $bin=/path/bin/hadoop\n/path/bin/hadoop). This jira is for common scripts. I filed a separate jira for HDFS scripts, which share the same pattern. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6997) 'hadoop' script should set LANG or LC_COLLATE explicitly for classpath order
[
https://issues.apache.org/jira/browse/HADOOP-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-6997.
--
Resolution: Not a Problem
Closing this as 'not a problem'. Classes should conflict in the class path.
'hadoop' script should set LANG or LC_COLLATE explicitly for classpath order
Key: HADOOP-6997
URL: https://issues.apache.org/jira/browse/HADOOP-6997
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 0.20.2, 0.21.0, 0.22.0
Reporter: Greg Roelofs
The 'hadoop' script builds the classpath in pieces, including the following
bit for the bulk of it:
{noformat}
# add libs to CLASSPATH
for f in $HADOOP_HOME/lib/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
{noformat}
The ordering of *.jar, i.e., the collation order, depends on either LANG or
LC_COLLATE on Linux systems. In the absence of either one, the script will
default to whatever the user's environment specifies; for Red Hat, the
default is en_US, which is a case-insensitive (and
punctuation-insensitive?) ordering. If LANG is set to C instead, the
ordering changes to the ASCII/UTF-8 byte ordering.
The key issue here is that $HADOOP_HOME/lib contains both upper- and
lowercase jar names (e.g., SimonTool.jar and commons-logging-1.1.1.jar,
to pick a completely random pair), which will have an inverted order
depending on which setting is used.
'hadoop' should explicitly set LANG and/or LC_COLLATE to whatever setting
it's implicitly assuming.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-7906) haoop-daemon.sh unconditionnally try to chown its log directory
[ https://issues.apache.org/jira/browse/HADOOP-7906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-7906. -- Resolution: Won't Fix This has been fixed as part of HADOOP-9902. haoop-daemon.sh unconditionnally try to chown its log directory --- Key: HADOOP-7906 URL: https://issues.apache.org/jira/browse/HADOOP-7906 Project: Hadoop Common Issue Type: Bug Components: scripts Affects Versions: 0.23.0, 0.23.1 Reporter: Bruno Mahé Labels: bigtop See: ./hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh:chown $HADOOP_IDENT_STRING $HADOOP_LOG_DIR In some setups, hadoop daemon may have the rights to write in its log dir but not to chown/chmod it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
Additionally, pid files are now created when daemons are run in interactive
mode. This will also prevent the accidental starting of two daemons with the
same configuration prior to launching java (i.e., fast fail without having to
wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly alterated, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS_*, YARN_*, and MAPRED_* environment variables act as overrides to
their equivalent HADOOP_* environment variables when 'hdfs', 'yarn', 'mapred',
and related commands are executed. Previously, these were separated out which
meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files via *_OPT.
* Some formerly 'documented' entries in yarn-env.sh have been undocumented as a
simple form of deprecration in order to greatly simplify configuration and
reduce unnecessary duplication. They will still work, but those variables will
likely be removed in a future release.
* Support for various undocumentented YARN log4j.properties files has been
removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented yarn.id.str has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP_SSH_PARALLEL} to prevent memory and network exhaustion.
By default, this is set to 10.
* ${HADOOP_CLIENT_OPTS} support has been added to a few more commands.
* Some subcommands were not listed in the usage.
* Various options on hadoop command lines were supported inconsistently. These
have been unified into hadoop-config.sh. --config is still required to be
first, however.
* ulimit logging for secure daemons no longer assumes /bin/bash but does assume
bash is on the command line path.
* Removed references to some Yahoo! specific paths.
* Removed unused slaves.sh from YARN build tree.
* Many exit states have been changed to reflect reality.
* Shell level errors now go to STDERR. Before, many of them went incorrectly
to STDOUT.
* CDPATH with a period (.) should no longer break the scripts.
* The scripts no longer try to chown directories.
IMPROVEMENTS:
* The *.out files are now appended instead of overwritten to allow for external
log rotation.
* The style and layout of the scripts is much more consistent across
subprojects.
* More of the shell code is now commented.
* Significant amounts of redundant code have been moved
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-11.patch #11 - This is essentially the same patch, but incorporates two trivial changes that I missed/forgot to fix: HADOOP-6746 and HADOOP-6851. The latter has been filed in different forms for several years under a variety of different JIRAs. It has also been recommended by a PMC member that I give this patch a week and if no issues pop up, to just commit it to trunk. So let's start the clock on that. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6863) Compile-native still fails on Mac OS X.
[ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6863. -- Resolution: Won't Fix lots has changed here. closing as stale. Compile-native still fails on Mac OS X. --- Key: HADOOP-6863 URL: https://issues.apache.org/jira/browse/HADOOP-6863 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.2.0 Reporter: Hong Tang Assignee: Hong Tang Attachments: hadoop-6863-20100715.patch, hadoop-6863-20100721.patch, hadoop-6863-20101105-1.patch I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold: - Although aclocal.m4 is changed after HADOOP-3659 is committed, the corresponding configure script is not re-generated. - It cannot find libjvm. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6865) there will be ant error if ant ran without network connected
[ https://issues.apache.org/jira/browse/HADOOP-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6865. -- Resolution: Won't Fix mavenization makes this stale. closing. there will be ant error if ant ran without network connected Key: HADOOP-6865 URL: https://issues.apache.org/jira/browse/HADOOP-6865 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 0.20.2 Environment: centos 5.4 Reporter: Evan Wang If you run `ant` without network connected, there will be an error below. And even if you connect your network, the error will exist. ivy-init-antlib: [typedef] java.util.zip.ZipException: error in opening zip file [typedef] at java.util.zip.ZipFile.open(Native Method) [typedef] at java.util.zip.ZipFile.init(ZipFile.java:114) [typedef] at java.util.zip.ZipFile.init(ZipFile.java:131) [typedef] at org.apache.tools.ant.AntClassLoader.getResourceURL(AntClassLoader.java:1028) [typedef] at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:147) [typedef] at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.init(AntClassLoader.java:109) [typedef] at org.apache.tools.ant.AntClassLoader.findResources(AntClassLoader.java:975) [typedef] at java.lang.ClassLoader.getResources(ClassLoader.java:1016) [typedef] at org.apache.tools.ant.taskdefs.Definer.resourceToURLs(Definer.java:364) [typedef] at org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:256) [typedef] at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) [typedef] at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) [typedef] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [typedef] at java.lang.reflect.Method.invoke(Method.java:597) [typedef] at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) [typedef] at org.apache.tools.ant.Task.perform(Task.java:348) [typedef] at org.apache.tools.ant.Target.execute(Target.java:357) [typedef] at org.apache.tools.ant.Target.performTasks(Target.java:385) [typedef] at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337) [typedef] at org.apache.tools.ant.Project.executeTarget(Project.java:1306) [typedef] at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41) [typedef] at org.apache.tools.ant.Project.executeTargets(Project.java:1189) [typedef] at org.apache.tools.ant.Main.runBuild(Main.java:758) [typedef] at org.apache.tools.ant.Main.startAnt(Main.java:217) [typedef] at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257) [typedef] at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104) [typedef] Could not load definitions from resource org/apache/ivy/ant/antlib.xml. It could not be found. BUILD FAILED /opt/hadoop-0.20.2/build.xml:1644: You need Apache Ivy 2.0 or later from http://ant.apache.org/ It could not be loaded from http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6874) JvmMetrics can't distinguish between jvms with same processName
[ https://issues.apache.org/jira/browse/HADOOP-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6874. -- Resolution: Incomplete I'm going to close this as stale, given metrics has been rewritten. If this is still an issue, please file a new Jira. Thanks! JvmMetrics can't distinguish between jvms with same processName --- Key: HADOOP-6874 URL: https://issues.apache.org/jira/browse/HADOOP-6874 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 0.20.2 Reporter: Franklin Ta Priority: Minor JvmMetrics has three tags: hostName, processName, and sessionId. For processes such as tasktracker/jobtracker/namenode/datanode which there is only one of on each host, these tags are fine. But for process names such as MAP and REDUCE, since there might be multiple jvms running map/reduce tasks, we might end up with multiple set of metrics which all have the same tags, and no way to find out which jvm they actually correspond to. (In addition, since there is jvm reuse, those process names might not correspond to the actual task being ran) A quick fix is to change this line in Child.java JvmMetrics.init(task.getPhase().toString(), job.getSessionId()); to this JvmMetrics.init(jvmId.toString(), job.getSessionId()); so that we are using the jvm id for the process name instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6894) Common foundation for Hadoop client tools
[ https://issues.apache.org/jira/browse/HADOOP-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6894. -- Resolution: Fixed This was more or less implemented, for better or worse. Closing as fixed. Common foundation for Hadoop client tools - Key: HADOOP-6894 URL: https://issues.apache.org/jira/browse/HADOOP-6894 Project: Hadoop Common Issue Type: New Feature Reporter: Alejandro Abdelnur Assignee: Owen O'Malley Attachments: HadoopClientTools.pdf, HadoopClientToolsV2.pdf, HadoopClientToolsV2_1.pdf, deployment.pdf As Hadoop widespreads and matures the number of tools and utilities for users keeps growing. Some of them are bundled with Hadoop core, some with Hadoop contrib, some on their own, some are full fledged servers on their own. For example, just to name a few: distcp, streaming, pipes, har, pig, hive, oozie. Today there is no standard mechanism for making these tools available to users. Neither there is a standard mechanism for these tools to integrate and distributed them with each other. The lack of a common foundation creates issues for developers and users. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-6916) Implement append operation for S3FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080191#comment-14080191 ] Allen Wittenauer commented on HADOOP-6916: -- Is this still an issue with all the S3 rework that happened? Implement append operation for S3FileSystem --- Key: HADOOP-6916 URL: https://issues.apache.org/jira/browse/HADOOP-6916 Project: Hadoop Common Issue Type: New Feature Components: fs/s3 Affects Versions: 0.20.2, 0.22.0 Reporter: Oleg Aleshko Priority: Minor Attachments: s3_append1.patch Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException(Not supported); S3FileSystem should be able to support appending, possibly via common block storage logic. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6918) Make metrics naming consistent
[
https://issues.apache.org/jira/browse/HADOOP-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-6918.
--
Resolution: Fixed
From comments, this appears to be fixed in 2.x, so closing.
Make metrics naming consistent
--
Key: HADOOP-6918
URL: https://issues.apache.org/jira/browse/HADOOP-6918
Project: Hadoop Common
Issue Type: Improvement
Components: metrics
Affects Versions: 0.20.2, 0.21.0
Reporter: Luke Lu
Assignee: Luke Lu
While working HADOOP-6728, I noticed that our metrics naming style is all
over the place:
* Capitalized camel case: e.g., FilesCreated in namenode metrics and some
rpc metrics
* uncapitalized camel case: e.g, threadsBlocked in jvm metrics and some rpc
metrics
* lowercased underscored: e.g., bytes_written in datanode metrics and
mapreduce metrics
Let's make them consistent. How about uncapitalized camel case? My main
reason for the camel case: some backends have limits on the name length and
underscore is wasteful.
Once we have a consistent naming style we can do:
@Metric(Number of INodes created) MutableCounterLong filesCreated;
instead of the more redundant:
@Metric({FilesCreated, Number of INodes created}) MutableCounterLong
filesCreated;
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-6937) Invalid Entry for JSP JARs in Hadoop CLASSPATH
[
https://issues.apache.org/jira/browse/HADOOP-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-6937.
--
Resolution: Incomplete
Likely stale.
Invalid Entry for JSP JARs in Hadoop CLASSPATH
--
Key: HADOOP-6937
URL: https://issues.apache.org/jira/browse/HADOOP-6937
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Ranjit Mathew
~/src/Hadoop/stage/hadoop-0.22.0-SNAPSHOT bin/hadoop classpath | sed
's/:/\n/g' |grep \\*
/home/ranjit/src/Hadoop/stage/hadoop-0.22.0-SNAPSHOT/bin/../lib/jsp-2.1/*.jar
The actual JSP JARs (jsp-2.1-6.1.14.jar jsp-api-2.1-6.1.14.jar) are just
under the lib folder - there
is no jsp-2.1 sub-folder.
The offending code is from bin/hadoop-config.sh:
156 for f in $HADOOP_COMMON_HOME/lib/jsp-2.1/*.jar; do
157 CLASSPATH=${CLASSPATH}:$f;
158 done
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-6959) Provisioning of long running Services via HOD
[ https://issues.apache.org/jira/browse/HADOOP-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6959. -- Resolution: Fixed Closing as Won't Fix. YARN-896 and associated JIRAs for a more modern take. Provisioning of long running Services via HOD - Key: HADOOP-6959 URL: https://issues.apache.org/jira/browse/HADOOP-6959 Project: Hadoop Common Issue Type: New Feature Environment: Should work on any environment Reporter: Saikat Kanjilal Priority: Minor Original Estimate: 60h Remaining Estimate: 60h Work on a computation model for services on the grid. The model would include: Various tools for defining clients and servers of the service, and at the least a C++ and Java instantiation of the abstractions Logical definitions of how to partition work onto a set of servers, i.e. a generalized shard implementation A few useful abstractions like locks (exclusive and RW, fairness), leader election, transactions, Various communication models for groups of servers belonging to a service, such as broadcast, unicast, etc. Tools for assuring QoS, reliability, managing pools of servers for a service with spares, etc. Integration with HDFS for persistence, as well as access to local filesystems Integration with ZooKeeper so that applications can use the namespace -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6961) Integration of Virtualization (such as Xen) with Hadoop tools
[ https://issues.apache.org/jira/browse/HADOOP-6961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6961. -- Resolution: Won't Fix I'm going to close this as Won't Fix. Here's why: With YARN (and even under 1.x), the task launcher stuff is pluggable. If one wanted to integrate KVM or some other virtualization support, the hooks are basically there now. Out of the box, cgroup support is there. If YARN-1964 moves forward, then Hadoop will have support for Docker-style containers as well. So this JIRA is better off being refiled as using a specific technology rather than something generic. Integration of Virtualization (such as Xen) with Hadoop tools - Key: HADOOP-6961 URL: https://issues.apache.org/jira/browse/HADOOP-6961 Project: Hadoop Common Issue Type: New Feature Environment: All Reporter: Saikat Kanjilal Priority: Minor How does one integrate sandboxing of arbitrary user code in C++ and other languages in a VM such as Xen with the Hadoop framework? How does this interact with SGE, Torque, Condor? As each individual machine has more and more cores/cpus, it makes sense to partition each machine into multiple virtual machines. That gives us a number of benefits: By assigning a virtual machine to a datanode, we effectively isolate the datanode from the load on the machine caused by other processes, making the datanode more responsive/reliable. With multiple virtual machines on each machine, we can lower the granularity of hod scheduling units, making it possible to schedule multiple tasktrackers on the same machine, improving the overall utilization of the whole clusters. With virtualization, we can easily snapshot a virtual cluster before releasing it, making it possible to re-activate the same cluster in the future and start to work from the snapshot. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-6962) FileSystem.mkdirs(Path, FSPermission) should use the permission for all of the created directories
[ https://issues.apache.org/jira/browse/HADOOP-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-6962: - Resolution: Fixed Status: Resolved (was: Patch Available) This appears to be working now on 2.x. Closing as fixed. FileSystem.mkdirs(Path, FSPermission) should use the permission for all of the created directories -- Key: HADOOP-6962 URL: https://issues.apache.org/jira/browse/HADOOP-6962 Project: Hadoop Common Issue Type: Bug Components: fs, security Affects Versions: 1.0.4 Reporter: Owen O'Malley Assignee: Daryn Sharp Priority: Blocker Labels: security Attachments: HADOOP-6962.patch Currently, FileSystem.mkdirs only applies the permissions to the last level if it was created. It should be applied to *all* levels that are created. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-6964) Allow compact property description in xml
[
https://issues.apache.org/jira/browse/HADOOP-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-6964:
-
Labels: newbie (was: )
Allow compact property description in xml
-
Key: HADOOP-6964
URL: https://issues.apache.org/jira/browse/HADOOP-6964
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Reporter: Owen O'Malley
Labels: newbie
We should allow users to use the more compact form of xml elements. For
example, we could allow:
{noformat}
property name=mapred.local.dir value=/disk/0/mapred,/disk/1/mapred/
{noformat}
The old format would also be supported.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-6967) docs update for -format command for namenode for Federation ( HDFS-1394 )
[ https://issues.apache.org/jira/browse/HADOOP-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-6967: - Component/s: documentation docs update for -format command for namenode for Federation ( HDFS-1394 ) - Key: HADOOP-6967 URL: https://issues.apache.org/jira/browse/HADOOP-6967 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Boris Shkolnik Assignee: Boris Shkolnik Labels: newbie new format for -format command is: namenode -format -clusterid cid commands_manual.html needs to be updated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-6967) docs update for -format command for namenode for Federation ( HDFS-1394 )
[ https://issues.apache.org/jira/browse/HADOOP-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-6967: - Labels: newbie (was: ) docs update for -format command for namenode for Federation ( HDFS-1394 ) - Key: HADOOP-6967 URL: https://issues.apache.org/jira/browse/HADOOP-6967 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Boris Shkolnik Assignee: Boris Shkolnik Labels: newbie new format for -format command is: namenode -format -clusterid cid commands_manual.html needs to be updated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-6974) Configurable header buffer size for Hadoop HTTP server
[ https://issues.apache.org/jira/browse/HADOOP-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-6974: - Labels: newbie (was: ) Configurable header buffer size for Hadoop HTTP server -- Key: HADOOP-6974 URL: https://issues.apache.org/jira/browse/HADOOP-6974 Project: Hadoop Common Issue Type: Improvement Reporter: Paul Butler Assignee: Paul Butler Labels: newbie Attachments: HADOOP-6974.3.patch, HADOOP-6974.4.patch, hadoop-6974.2.patch, hadoop-6974.patch This patch adds a configurable parameter dfs.http.header.buffer.size to Hadoop which allows the buffer size to be configured from the xml configuration. This fixes an issue that came up in an environment where the Hadoop servers share a domain with other web applications that use domain cookies. The large cookies overwhelmed Jetty's buffer which caused it to return a 413 error. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6981) Provide a way to run system tests outside of source tree and Ant build environment
[ https://issues.apache.org/jira/browse/HADOOP-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6981. -- Resolution: Incomplete Closing as stale. Provide a way to run system tests outside of source tree and Ant build environment -- Key: HADOOP-6981 URL: https://issues.apache.org/jira/browse/HADOOP-6981 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 0.22.0 Reporter: Konstantin Boudnik In the current implementation the only way to execute Herriot (system) tests is to run certain Ant build target. As the result every test run causes full recompilation and re-weaving of AOP code. It takes about 2 minutes for 0.20 code and 1m20s for 0.21+ However, on multiple runs of the same test the waste of time might be significant. It'd be very helpful to be able to just run the tests our of test jar file or something without being force to recompile the code again and again. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6983) Update balancer docs
[ https://issues.apache.org/jira/browse/HADOOP-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6983. -- Resolution: Incomplete Closing as stale. Update balancer docs Key: HADOOP-6983 URL: https://issues.apache.org/jira/browse/HADOOP-6983 Project: Hadoop Common Issue Type: Improvement Reporter: Dmytro Molkov Assignee: Dmytro Molkov Attachments: HADOOP-6983.patch Once HDFS-1105 gets commited we will need to update the docs for the balancer command line. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081131#comment-14081131 ] Allen Wittenauer commented on HADOOP-9902: -- bq. Should we add the descriptions? We should. I've been waiting for someone to notice. I put it that way (or another) almost immediately after the code that added that option got committed to trunk. That was about a month or so ago. bq. I think we ought to consider a strategy of get it in trunk and then evolve it until happy, finally backporting to branch-2. I've had a lot of discussions with a lot of committers over the past year regarding this change. (Including a significant amount of begging to get a review done). To me, pushing this to trunk where end user impact is still extremely low, incompatible changes are OK, and gives the community a chance to play is really the only viable strategy to build confidence with the vast majority of the committership, review or otherwise. It's unfortunately a large patch. It's unfortunately impractical to split apart due to the nature of the *current* code base (as soon as you touch hadoop-config.sh or *-env.sh, you impact _everything_ ...) . Realistically, there will be breakage. It's an unattainable goal to move this code forward in a significant way while maintaining 100% compatibility. It's one of the reasons why I changed reset my personal goal to get it just into trunk rather than branch-2. The best we can do is fix or document those cases we identify that will break. The worst is to maintain the status quo, continue ignoring the operational issues the current shell code inflicts, digging the hole deeper with every change that we make (.e.g., YARN-1429). bq. have you a branch on github I can merge I did at one point, but I gave up. There is no question this is a large patch covering a lot of ground. The manual merges every few weeks with every sub command that got added or every minor tweak got to be too much, esp for a branch where I would be the only one paying attention. (As a result, I feel as though I could have a lovely conversation with Sisyphus.) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6999) security implementation for new FileSystem (FileContext) API
[ https://issues.apache.org/jira/browse/HADOOP-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6999. -- Resolution: Fixed Likely stale. security implementation for new FileSystem (FileContext) API Key: HADOOP-6999 URL: https://issues.apache.org/jira/browse/HADOOP-6999 Project: Hadoop Common Issue Type: Improvement Reporter: Krishna Ramachandran New file system API (HADOOP 4952) should implement security features currently provided by FileSystem APIs This is a critical requirement for MapReduce components to migrate and use new APIs for internal filesystem operations (MAPREDUCE 2020) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-7092) Consolidate startup shell scripts
[ https://issues.apache.org/jira/browse/HADOOP-7092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-7092. -- Resolution: Won't Fix Startup scripts are being collected in Big Top. Closing. Consolidate startup shell scripts - Key: HADOOP-7092 URL: https://issues.apache.org/jira/browse/HADOOP-7092 Project: Hadoop Common Issue Type: Task Components: scripts Reporter: Thomas Koch Priority: Minor - Today we struggled ~1-2 hours to find out that the Cloudera init scripts have a bug in the shell code (bug filled by my colleague) - Many projects that started at hadoop copied and adapted the shell startup code. So there's a lot of code duplication between hbase, zookeeper, hadoop and maybe others - There already are some open issues regarding to shell code - The shell code isn't properly tested (automaticaly) and will first probably fail at customer side Would it make sense to build a shell library of most often used functionality for startup scripts? Is there already such a library somewhere? This issue should collect thoughts in this area. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-7020) establish a Powered by Hadoop logo
[ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-7020. -- Resolution: Fixed Thanks to [~owen.omalley] for pointing out the following: http://mail-archives.apache.org/mod_mbox/hadoop-general/201106.mbox/%3c8f5720e1-00d0-4352-b9d4-76c51c48f...@apache.org%3E ... http://mail-archives.apache.org/mod_mbox/hadoop-general/201106.mbox/%3cbab91a74-e0cc-4a65-a0ac-4186014e4...@apache.org%3E So closing. establish a Powered by Hadoop logo Key: HADOOP-7020 URL: https://issues.apache.org/jira/browse/HADOOP-7020 Project: Hadoop Common Issue Type: Improvement Components: documentation Affects Versions: site Reporter: Doug Cutting Assignee: Doug Cutting Fix For: site Attachments: PoweredByApacheHadoop.png, PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, hadoop.JPG, pbh-64-logos.png, powered-by-apache-hadoop.png, powered-by-hadoop-small.png, powered-by-hadoop.png, powered-by-proposal.jpg, powered_by_hadoop.jpg We should agree on a Powered By Hadoop logo, as suggested in: http://www.apache.org/foundation/marks/pmcs#poweredby -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9873) hadoop-env.sh got called multiple times
[
https://issues.apache.org/jira/browse/HADOOP-9873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9873:
-
Resolution: Duplicate
Status: Resolved (was: Patch Available)
This is part of HADOOP-9902. Closing as a dupe.
hadoop-env.sh got called multiple times
---
Key: HADOOP-9873
URL: https://issues.apache.org/jira/browse/HADOOP-9873
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 3.0.0
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
Attachments: HADOOP-9873.patch
Ref. below, it can be seen hadoop-env.sh got called multiple times when
running something like 'hadoop-daemon.sh start namenode'.
{noformat}
[drankye@zkdev ~]$ cd $HADOOP_PREFIX
[drankye@zkdev hadoop-3.0.0-SNAPSHOT]$ grep -r hadoop-env *
libexec/hadoop-config.sh:if [ -e ${HADOOP_PREFIX}/conf/hadoop-env.sh ]; then
libexec/hadoop-config.sh:if [ -f ${HADOOP_CONF_DIR}/hadoop-env.sh ]; then
libexec/hadoop-config.sh: . ${HADOOP_CONF_DIR}/hadoop-env.sh
sbin/hadoop-daemon.sh:if [ -f ${HADOOP_CONF_DIR}/hadoop-env.sh ]; then
sbin/hadoop-daemon.sh: . ${HADOOP_CONF_DIR}/hadoop-env.sh
{noformat}
Considering the following lines in hadoop-env.sh
{code}
# Command specific options appended to HADOOP_OPTS when specified
export
HADOOP_NAMENODE_OPTS=-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS}
-Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender}
$HADOOP_NAMENODE_OPTS
{code}
It may end with some redundant result like below when called multiple times.
{noformat}
HADOOP_NAMENODE_OPTS='-Dhadoop.security.logger=INFO,RFAS
-Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS
-Dhdfs.audit.logger=INFO,NullAppender '
{noformat}
It's not a big issue for now however it would be better to be clean and avoid
this since it can cause the final JAVA command line is very lengthy and hard
to read.
A possible fix would be to add a flag variable like HADOOP_ENV_INITED in
hadoop-env.sh, and then at the beginning of it check the flag. If the flag
evaluates true, then return immediately.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10060) Unable to run JDK_1.7.1_40 commands in Cygwin2.829 (32 bit
[ https://issues.apache.org/jira/browse/HADOOP-10060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10060: -- Priority: Major (was: Blocker) Target Version/s: (was: 1.2.1) Fix Version/s: (was: 1.2.1) Unable to run JDK_1.7.1_40 commands in Cygwin2.829 (32 bit -- Key: HADOOP-10060 URL: https://issues.apache.org/jira/browse/HADOOP-10060 Project: Hadoop Common Issue Type: Task Components: bin Affects Versions: 1.2.1 Environment: Background I have installed Cygwin to self learn Hadoop 1.2.1 with the above JDK in Cygwin environment on Windows XP- SP2 laptop Reporter: Anand Murali Labels: test Actions Performed 1. As advised by Hadoop exported JAVA_HOME=/cygwin/jdk1.2.7_40/bin in hadoop-env.sh and tried to run java commands with error jar /cygwin/hadoop-1.2.1/hadoop-examples-1.2.1.jar -bash: jar: command not found 2. However when echoed I get right pointed path echo $JAVA_HOME /cygwin/jdk1.7.0_40/bin 3. Tried including export path in .bash_profile, and .profile, with same error. Unable to move forward with out JDK calls. Anticipate the same with Hadoop related commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10060) Unable to run JDK_1.7.1_40 commands in Cygwin2.829 (32 bit
[ https://issues.apache.org/jira/browse/HADOOP-10060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10060: -- Labels: (was: \ test) Unable to run JDK_1.7.1_40 commands in Cygwin2.829 (32 bit -- Key: HADOOP-10060 URL: https://issues.apache.org/jira/browse/HADOOP-10060 Project: Hadoop Common Issue Type: Task Components: bin Affects Versions: 1.2.1 Environment: Background I have installed Cygwin to self learn Hadoop 1.2.1 with the above JDK in Cygwin environment on Windows XP- SP2 laptop Reporter: Anand Murali Actions Performed 1. As advised by Hadoop exported JAVA_HOME=/cygwin/jdk1.2.7_40/bin in hadoop-env.sh and tried to run java commands with error jar /cygwin/hadoop-1.2.1/hadoop-examples-1.2.1.jar -bash: jar: command not found 2. However when echoed I get right pointed path echo $JAVA_HOME /cygwin/jdk1.7.0_40/bin 3. Tried including export path in .bash_profile, and .profile, with same error. Unable to move forward with out JDK calls. Anticipate the same with Hadoop related commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-9521) krb5 replay error triggers log file DoS with Safari
[ https://issues.apache.org/jira/browse/HADOOP-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-9521. -- Resolution: Won't Fix krb5 replay error triggers log file DoS with Safari --- Key: HADOOP-9521 URL: https://issues.apache.org/jira/browse/HADOOP-9521 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.0.4-alpha Environment: Mac OS X 10.8.3, Safari 6.0.3 (8536.28.10) Mac OS X 10.6.8, Safari 6.0.3 (8536.28.10) Reporter: Allen Wittenauer Priority: Blocker While investigating YARN-621, looking at the web interface with Safari triggered a loop which both filled the log with stack traces as well as left the browser in a continual loading situation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-8777) Retrieve job id on execution of a job
[ https://issues.apache.org/jira/browse/HADOOP-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-8777: - Priority: Major (was: Blocker) Retrieve job id on execution of a job - Key: HADOOP-8777 URL: https://issues.apache.org/jira/browse/HADOOP-8777 Project: Hadoop Common Issue Type: New Feature Reporter: Nelson Paul A method to retrieve the job id on submitting the job (using a JAVA client program) will do a lot. It will be easier to track the job if the job id and job can be linked in some way. Currently there is no direct way to identify the job id of a particular job. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10060) Unable to run JDK_1.7.1_40 commands in Cygwin2.829 (32 bit
[ https://issues.apache.org/jira/browse/HADOOP-10060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10060: -- Labels: \ test (was: test) Unable to run JDK_1.7.1_40 commands in Cygwin2.829 (32 bit -- Key: HADOOP-10060 URL: https://issues.apache.org/jira/browse/HADOOP-10060 Project: Hadoop Common Issue Type: Task Components: bin Affects Versions: 1.2.1 Environment: Background I have installed Cygwin to self learn Hadoop 1.2.1 with the above JDK in Cygwin environment on Windows XP- SP2 laptop Reporter: Anand Murali Actions Performed 1. As advised by Hadoop exported JAVA_HOME=/cygwin/jdk1.2.7_40/bin in hadoop-env.sh and tried to run java commands with error jar /cygwin/hadoop-1.2.1/hadoop-examples-1.2.1.jar -bash: jar: command not found 2. However when echoed I get right pointed path echo $JAVA_HOME /cygwin/jdk1.7.0_40/bin 3. Tried including export path in .bash_profile, and .profile, with same error. Unable to move forward with out JDK calls. Anticipate the same with Hadoop related commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-8777) Retrieve job id on execution of a job
[ https://issues.apache.org/jira/browse/HADOOP-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-8777. -- Resolution: Incomplete Likely stale. Retrieve job id on execution of a job - Key: HADOOP-8777 URL: https://issues.apache.org/jira/browse/HADOOP-8777 Project: Hadoop Common Issue Type: New Feature Reporter: Nelson Paul A method to retrieve the job id on submitting the job (using a JAVA client program) will do a lot. It will be easier to track the job if the job id and job can be linked in some way. Currently there is no direct way to identify the job id of a particular job. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-8610) test-patch should run tests in the root repo
[ https://issues.apache.org/jira/browse/HADOOP-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-8610. -- Resolution: Incomplete Likely stale. test-patch should run tests in the root repo Key: HADOOP-8610 URL: https://issues.apache.org/jira/browse/HADOOP-8610 Project: Hadoop Common Issue Type: Improvement Reporter: Eli Collins Priority: Blocker The test patch target should run tests from root projets, eg hadoop-tools. Otherwise we miss patches that introduce test failures, eg HDFS-3690. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-8139) Path does not allow metachars to be escaped
[
https://issues.apache.org/jira/browse/HADOOP-8139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-8139:
-
Priority: Major (was: Blocker)
Path does not allow metachars to be escaped
---
Key: HADOOP-8139
URL: https://issues.apache.org/jira/browse/HADOOP-8139
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 0.23.3, 3.0.0
Reporter: Daryn Sharp
Attachments: HADOOP-8139-2.patch, HADOOP-8139-3.patch,
HADOOP-8139-4.patch, HADOOP-8139-5.patch, HADOOP-8139-6.patch,
HADOOP-8139.patch, HADOOP-8139.patch
Path converts \ into /, probably for windows support? This means it's
impossible for the user to escape metachars in a path name. Glob expansion
can have deadly results.
Here are the most egregious examples. A user accidentally creates a path like
/user/me/*/file. Now they want to remove it.
{noformat}hadoop fs -rmr -skipTrash '/user/me/\*' becomes...
hadoop fs -rmr -skipTrash /user/me/*{noformat}
* User/Admin: Nuked their home directory or any given directory
{noformat}hadoop fs -rmr -skipTrash '\*' becomes...
hadoop fs -rmr -skipTrash /*{noformat}
* User: Deleted _everything_ they have access to on the cluster
* Admin: *Nukes the entire cluster*
Note: FsShell is shown for illustrative purposes, however the problem is in
the Path object, not FsShell.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-7035) Document incompatible API changes between releases
[ https://issues.apache.org/jira/browse/HADOOP-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-7035: - Fix Version/s: (was: 0.22.1) Document incompatible API changes between releases -- Key: HADOOP-7035 URL: https://issues.apache.org/jira/browse/HADOOP-7035 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Tom White Assignee: Tom White Attachments: apicheck-hadoop-0.20.203.0-0.20.204.0.txt, apicheck-hadoop-0.21.0-0.22.0-SNAPSHOT.txt, jdiff-with-previous-release.sh, jdiff-with-previous-release.sh We can use JDiff to generate a list of incompatible changes for each release. See https://issues.apache.org/jira/browse/HADOOP-6668?focusedCommentId=12860017page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12860017 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-7035) Document incompatible API changes between releases
[ https://issues.apache.org/jira/browse/HADOOP-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-7035: - Priority: Major (was: Blocker) Document incompatible API changes between releases -- Key: HADOOP-7035 URL: https://issues.apache.org/jira/browse/HADOOP-7035 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Tom White Assignee: Tom White Attachments: apicheck-hadoop-0.20.203.0-0.20.204.0.txt, apicheck-hadoop-0.21.0-0.22.0-SNAPSHOT.txt, jdiff-with-previous-release.sh, jdiff-with-previous-release.sh We can use JDiff to generate a list of incompatible changes for each release. See https://issues.apache.org/jira/browse/HADOOP-6668?focusedCommentId=12860017page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12860017 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-7197) Support for non-standard ssh port for slaves
[ https://issues.apache.org/jira/browse/HADOOP-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-7197. -- Resolution: Won't Fix Support for non-standard ssh port for slaves Key: HADOOP-7197 URL: https://issues.apache.org/jira/browse/HADOOP-7197 Project: Hadoop Common Issue Type: New Feature Components: util Affects Versions: 0.20.2 Environment: FreeBSD 8.2-RELEASE Reporter: Vrachnis Ilias-Dimitrios Priority: Trivial Labels: hadoop Attachments: slaves.sh.patch I was trying to add a slave that ran sshd in a non-standard port (eg. in stead of 22), when I noticed that there was no way to support another port through the configuration for a single node. Supporting a different port for all the slaves is possible through the HADOOP_SSH_OPTS variable, but not for a single slave. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-9777) RPM should not claim ownership of paths owned by the platform
[ https://issues.apache.org/jira/browse/HADOOP-9777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-9777. -- Resolution: Won't Fix Closing as won't fix, since we no longer build RPMs as part of the main Hadoop project. RPM should not claim ownership of paths owned by the platform - Key: HADOOP-9777 URL: https://issues.apache.org/jira/browse/HADOOP-9777 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1.1.2 Environment: Fedora 19 x64 Reporter: Stevo Slavic Priority: Critical Installing Apache Hadoop rpm ( hadoop-1.1.2-1.x86_64.rpm ) on Fedora 19 x64 fails with: [root@laptop hadoop]# rpm -i /home/sslavic/Downloads/hadoop-1.1.2-1.x86_64.rpm file /usr/bin from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 file /usr/lib from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 file /usr/lib64 from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 file /usr/sbin from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 Same issue occurs if one tries to install as non-root user: [sslavic@laptop ~]$ sudo rpm -i Downloads/hadoop-1.1.2-1.x86_64.rpm file /usr/bin from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 file /usr/lib from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 file /usr/lib64 from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 file /usr/sbin from install of hadoop-1.1.2-1.x86_64 conflicts with file from package filesystem-3.2-12.fc19.x86_64 It seems these 4 directories in Hadoop rpm have wrong permissions (+w for owner). This is violation of packaging rules. Hadoop rpm spec and/or build scripts need to be fixed, so that rpm on installation doesn't try to claim ownership of paths owned by the platform, in this case, filesystem. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9590) Move to JDK7 improved APIs for file operations when available
[ https://issues.apache.org/jira/browse/HADOOP-9590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081565#comment-14081565 ] Allen Wittenauer commented on HADOOP-9590: -- This was recently discussed in common-dev, but I don't remember the outcome. Move to JDK7 improved APIs for file operations when available - Key: HADOOP-9590 URL: https://issues.apache.org/jira/browse/HADOOP-9590 Project: Hadoop Common Issue Type: Improvement Reporter: Ivan Mitic JDK6 does not have a complete support for local file system file operations. Specifically: - There is no symlink/hardlink APIs what forced Hadoop to defer to shell based tooling - No error information returned when File#mkdir/mkdirs or File#renameTo fails, making it unnecessary hard to troubleshoot some issues - File#canRead/canWrite/canExecute do not perform any access checks on Windows making APIs inconsistent with the Unix behavior - File#setReadable/setWritable/setExecutable do not change access rights on Windows making APIs inconsistent with the Unix behavior - File#length does not work as expected on symlinks on Windows - File#renameTo does not work as expected on symlinks on Windows All above resulted in Hadoop community having to fill in the gaps by providing equivalent native implementations or applying workarounds. JDK7 addressed (as far as I know) all (or most) of the above problems, either thru the newly introduced [Files|http://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html] class or thru bug fixes. This is a tracking Jira to revisit above mediations once JDK7 becomes the supported platform by the Hadoop community. This work would allow significant portion of the native platform-dependent code to be replaced with Java equivalents what is goodness w.r.t. Hadoop cross-platform support. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-9520) _HOST doesn't resolve to bound interface
[ https://issues.apache.org/jira/browse/HADOOP-9520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-9520. -- Resolution: Won't Fix _HOST doesn't resolve to bound interface Key: HADOOP-9520 URL: https://issues.apache.org/jira/browse/HADOOP-9520 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.4-alpha Reporter: Allen Wittenauer _HOST appears to ignore bound interfaces. For example, if a host has two interfaces such that: nic0 = gethostname() nic1 = someothername and then I configure the namenode or resource manager to use someothername:, the system still treats _HOST = nic0. This is especially harmful for Kerberos principals. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9082) Simplify scripting usages so that parallel platform-dependent scripts are simple to maintain
[ https://issues.apache.org/jira/browse/HADOOP-9082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081586#comment-14081586 ] Allen Wittenauer commented on HADOOP-9082: -- It's interesting reading this thread in light of my own attempt to greatly improve the bash code in HADOOP-9902. Simplify scripting usages so that parallel platform-dependent scripts are simple to maintain Key: HADOOP-9082 URL: https://issues.apache.org/jira/browse/HADOOP-9082 Project: Hadoop Common Issue Type: Bug Reporter: Matt Foley This issue was formerly titled Select and document a platform-independent scripting language for use in Hadoop environment and was discussed in the common-dev@ threads [PROPOSAL] introduce Python as build-time and run-time dependency for Hadoop and throughout Hadoop stack and [VOTE] introduce Python as build-time and run-time dependency for Hadoop and throughout Hadoop stack. In that discussion the community consensus rejected the idea of using Python as a cross-platform scripting language. It is now proposed to follow up Allen's suggestions below about simplifying the use of scripts in Hadoop, bring more functionality into core code, and try to leave us with only trivial usages of scripting that can readily be maintained in as many platform-specific scripting languages as necessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-8797) automatically detect JAVA_HOME on Linux, report native lib path similar to class path
[ https://issues.apache.org/jira/browse/HADOOP-8797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-8797. -- Resolution: Duplicate hadoop jnipath is now part of HADOOP-9902. Closing. automatically detect JAVA_HOME on Linux, report native lib path similar to class path - Key: HADOOP-8797 URL: https://issues.apache.org/jira/browse/HADOOP-8797 Project: Hadoop Common Issue Type: Improvement Environment: Linux Reporter: Gera Shegalov Priority: Trivial Attachments: HADOOP-8797.patch Enhancement 1) iterate common java locations on Linux starting with Java7 down to Java6 Enhancement 2) hadoop jnipath to print java.library.path similar to hadoop classpath -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-8187) Improve the discovery of the jvm library during the build process
[ https://issues.apache.org/jira/browse/HADOOP-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-8187. -- Resolution: Won't Fix Lack of consensus. Closing as won't fix. Improve the discovery of the jvm library during the build process - Key: HADOOP-8187 URL: https://issues.apache.org/jira/browse/HADOOP-8187 Project: Hadoop Common Issue Type: Improvement Reporter: Devaraj Das Improve the discovery of the jvm library during the build of native libraries/libhdfs/fuse-dfs, etc. A couple of different ways are currently used (discussed in HADOOP-6924). We should clean this part up and also consider builds of native stuff on OSX. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-8229) DistCp doesn't handle non-existent paths correctly
[
https://issues.apache.org/jira/browse/HADOOP-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-8229.
--
Resolution: Won't Fix
distcpv1 was replaced by distcpv2. Closing as won't fix.
DistCp doesn't handle non-existent paths correctly
--
Key: HADOOP-8229
URL: https://issues.apache.org/jira/browse/HADOOP-8229
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.20.2, 1.0.0
Reporter: Jakob Homan
Assume /user/jhoman/blork doesn't exist:
{noformat}[tardis]$ hadoop distcp 'hftp://nn:50070/user/jhoman/blork'
/tmp/plork
12/03/29 22:04:33 INFO tools.DistCp:
srcPaths=[hftp://nn:50070/user/jhoman/blork]
12/03/29 22:04:33 INFO tools.DistCp: destPath=/tmp/plork
[Fatal Error] :1:173: XML document structures must start and end within the
same entity.
With failures, global counters are inaccurate; consider running with -i
Copy failed: java.io.IOException: invalid xml directory content
at
org.apache.hadoop.hdfs.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:427)
at
org.apache.hadoop.hdfs.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:432)
at
org.apache.hadoop.hdfs.HftpFileSystem.getFileStatus(HftpFileSystem.java:461)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
at org.apache.hadoop.tools.DistCp.checkSrcPath(DistCp.java:636)
at org.apache.hadoop.tools.DistCp.copy(DistCp.java:656)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
Caused by: org.xml.sax.SAXParseException: XML document structures must start
and end within the same entity.
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1231)
at
org.apache.hadoop.hdfs.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:421)
... 9 more{noformat}
This is because the ListPathsServlet hits an NPE when it calls
nnproxy.getFileInfo(path); on the non-existent path and just bails, leaving
the resulting XML unformed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-7462) Feature 'http://apache.org/xml/features/xinclude' is not recognized error in Hadoop 0.20.2
[ https://issues.apache.org/jira/browse/HADOOP-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-7462. -- Resolution: Won't Fix Stale. Won't fix. Feature 'http://apache.org/xml/features/xinclude' is not recognized error in Hadoop 0.20.2 -- Key: HADOOP-7462 URL: https://issues.apache.org/jira/browse/HADOOP-7462 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 0.20.2 Environment: Ubunto Linux Java 1.6.0_24 Reporter: fodil I have build a library to manipulate Hbase tables and am calling these methods inside a complexe java project. I am using the compiled version of Hadoop 0.20.2 and hbase 0.90.3. My project compiled without any problem, during the execution amd having the error to xInclude feature is not recognized. I have found suggestion to put a patch in configuration.java, but the issue am using the compiled version of hadoop. below is the stack of the error. thanks for your help Jul 15, 2011 10:10:11 AM org.apache.hadoop.conf.Configuration loadResource SEVERE: error parsing conf file: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1171) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1030) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:980) at org.apache.hadoop.conf.Configuration.get(Configuration.java:382) at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:63) at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:89) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:100) at org.b3mn.poem.Hbase_hadoop_data_access_methods.addRow(Hbase_hadoop_data_access_methods.java:46) at org.b3mn.poem.Representation.contentExists(Representation.java:108) at org.b3mn.poem.Representation.setSvg(Representation.java:239) at org.b3mn.poem.Identity.newModel(Identity.java:92) at org.b3mn.poem.handler.NewModelHandler.doPost(NewModelHandler.java:129) at org.b3mn.poem.Dispatcher.dispatch(Dispatcher.java:376) at org.b3mn.poem.Dispatcher.doPost(Dispatcher.java:409) at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.b3mn.poem.security.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:156) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:662) Caused by: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1061) ... 30 more -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-7459) rpm shouldn't include jdk dependency check
[ https://issues.apache.org/jira/browse/HADOOP-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-7459. -- Resolution: Won't Fix Won't fix. We no longer ship RPMs. Closing. rpm shouldn't include jdk dependency check -- Key: HADOOP-7459 URL: https://issues.apache.org/jira/browse/HADOOP-7459 Project: Hadoop Common Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: no-jdk.patch There are many ways to install java and the rpm shouldn't insist on one method. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-584) Calling shell scripts from build.xml discriminates Windows user minority.
[ https://issues.apache.org/jira/browse/HADOOP-584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-584. - Resolution: Fixed Calling shell scripts from build.xml discriminates Windows user minority. - Key: HADOOP-584 URL: https://issues.apache.org/jira/browse/HADOOP-584 Project: Hadoop Common Issue Type: Bug Components: scripts Affects Versions: 0.7.0 Environment: Windows Reporter: Konstantin Shvachko This is introduced by HADOOP-567. The problem is that now I cannot even build hadoop in Eclipse under Windows unless I run it under Cygwin. This is in a way the same as calling make in build.xml, which was recently fixed HADOOP-537. I think we should not introducing more dependencies on Cygwin just in order to show something in Web UI. I also don't remember we claimed that Cygwin or anything else except for Ant is required for Hadoop builds. Is there another way of solving this? build.xml defines version property, Ant has user.name property. URL is not changing very often. Or may be the web ui should obtain these properties in run-time. Or may be the Packaging is a better solution, as you guys discussed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10935) Cleanup HadoopKerberosName for public consumption
Allen Wittenauer created HADOOP-10935: - Summary: Cleanup HadoopKerberosName for public consumption Key: HADOOP-10935 URL: https://issues.apache.org/jira/browse/HADOOP-10935 Project: Hadoop Common Issue Type: Bug Reporter: Allen Wittenauer Priority: Minor It would be good if we pulled HadoopKerberosName out of the closet and into the light so that others may bask in its glorious usefulness. Missing: * Documentation * Shell short cut * CLI help when run without arguments -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10935) Cleanup HadoopKerberosName for public consumption
[
https://issues.apache.org/jira/browse/HADOOP-10935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085036#comment-14085036
]
Allen Wittenauer commented on HADOOP-10935:
---
For those that aren't aware, the HadoopKerberosName class has a main method
that allows one an easy way to play with the auth_to_local rules. For probably
99% of the universe, this isn't really necessary, but for the 1% it's awesome.
{code}
$ bin/hadoop org.apache.hadoop.security.HadoopKerberosName a...@altiscale.com
Name: a...@altiscale.com to aw
{code}
Cleanup HadoopKerberosName for public consumption
-
Key: HADOOP-10935
URL: https://issues.apache.org/jira/browse/HADOOP-10935
Project: Hadoop Common
Issue Type: Bug
Reporter: Allen Wittenauer
Priority: Minor
It would be good if we pulled HadoopKerberosName out of the closet and into
the light so that others may bask in its glorious usefulness.
Missing:
* Documentation
* Shell short cut
* CLI help when run without arguments
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10935) Cleanup HadoopKerberosName for public consumption
[ https://issues.apache.org/jira/browse/HADOOP-10935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10935: -- Component/s: documentation Labels: newbie (was: ) Issue Type: Improvement (was: Bug) Cleanup HadoopKerberosName for public consumption - Key: HADOOP-10935 URL: https://issues.apache.org/jira/browse/HADOOP-10935 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Allen Wittenauer Priority: Minor Labels: newbie It would be good if we pulled HadoopKerberosName out of the closet and into the light so that others may bask in its glorious usefulness. Missing: * Documentation * Shell short cut * CLI help when run without arguments -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-12.patch rebase for git rev d469993fe9b270d01c4cf4542bb701297996fbac Minor stuff: * add usage for keys, credential * dirs with spaces should work again (this is very very broken in trunk btw) * merged in hadoop classpath changes from HADOOP-10903 * probably worthwhile to point out that this repairs the defaults that HADOOP-10759 broke ... will commit on Friday. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
Additionally, pid files are now created when daemons are run in interactive
mode. This will also prevent the accidental starting of two daemons with the
same configuration prior to launching java (i.e., fast fail without having to
wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly alterated, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS_*, YARN_*, and MAPRED_* environment variables act as overrides to
their equivalent HADOOP_* environment variables when 'hdfs', 'yarn', 'mapred',
and related commands are executed. Previously, these were separated out which
meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files via *_OPT.
* Some formerly 'documented' entries in yarn-env.sh have been undocumented as a
simple form of deprecration in order to greatly simplify configuration and
reduce unnecessary duplication. They will still work, but those variables will
likely be removed in a future release.
* Support for various undocumentented YARN log4j.properties files has been
removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented yarn.id.str has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP_SSH_PARALLEL} to prevent memory and network exhaustion.
By default, this is set to 10.
* ${HADOOP_CLIENT_OPTS} support has been added to a few more commands.
* Some subcommands were not listed in the usage.
* Various options on hadoop command lines were supported inconsistently. These
have been unified into hadoop-config.sh. --config is still required to be
first, however.
* ulimit logging for secure daemons no longer assumes /bin/bash but does assume
bash is on the command line path.
* Removed references to some Yahoo! specific paths.
* Removed unused slaves.sh from YARN build tree.
* Many exit states have been changed to reflect reality.
* Shell level errors now go to STDERR. Before, many of them went incorrectly
to STDOUT.
* CDPATH with a period (.) should no longer break the scripts.
* The scripts no longer try to chown directories.
* Hadoop deployments where the directories have spaces should work again.
IMPROVEMENTS:
* The *.out files are now appended instead of overwritten to allow for external
log rotation.
* The style and layout of the scripts is much more consistent across
subprojects.
* More of the shell code is
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085537#comment-14085537 ] Allen Wittenauer commented on HADOOP-9902: -- FWIW, I asked Roman (in person even!) over a month ago to look at it. So what do you view as a reasonable timeline to wait [~andrew.wang]? Another month? Two? Or is the clock based on rebasing? Have *you* even tried it? Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085664#comment-14085664 ] Allen Wittenauer commented on HADOOP-9902: -- bq. but I think Allen will admit that he has not been pushing on it hard until rather recently. The second rev of the patch was only posted at the beginning of last month. There was certainly a pause between Oct and Jan, but it should be pointed out that I deleted previous versions of the patch, in particular the ones not in patch format and ones against branch-2. (Sort by date helps here.) The main chunk of the code has been finished since September. bq. Many or most of the Hadoop committers are less familiar with shell scripting than they are with other types of programming, and so don't feel well qualified to do so. This begs the question: who has been reviewing all the new shell code that is going in then? (We've had entire new scripts thrown in from all over, not just minor stuff!) From the basis of this thread, it sounds like the only people who should be reviewing shell code is Roman and (I guess) myself. Yet I haven't been asked to look at any in years... Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-13.patch HADOOP-9902-13-branch-2.patch Rebase for current and fixes a bug introduced by HADOOP-10927. Additionally, it was requested I provide a branch-2 patch for people to try out. This was a 'lazy' backport: not well tested and probably includes commands that don't exist in branch-2. Clearly not meant to be committed. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
