[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Open (was: Patch Available) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Patch Available (was: Open) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-16.patch -16: re-deprecate the previously not deprecated but documented hadoop nfs3 and hadoop portmap subcommands Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-9870) Mixed configurations for JVM -Xmx in hadoop command
[
https://issues.apache.org/jira/browse/HADOOP-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-9870.
--
Resolution: Duplicate
Closing this as HADOOP-9902 contains a fix for this issue.
Mixed configurations for JVM -Xmx in hadoop command
---
Key: HADOOP-9870
URL: https://issues.apache.org/jira/browse/HADOOP-9870
Project: Hadoop Common
Issue Type: Bug
Reporter: Wei Yan
Attachments: HADOOP-9870.patch, HADOOP-9870.patch, HADOOP-9870.patch
When we use hadoop command to launch a class, there are two places setting
the -Xmx configuration.
*1*. The first place is located in file
{{hadoop-common-project/hadoop-common/src/main/bin/hadoop}}.
{code}
exec $JAVA $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS $@
{code}
Here $JAVA_HEAP_MAX is configured in hadoop-config.sh
({{hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh}}). The
default value is -Xmx1000m.
*2*. The second place is set with $HADOOP_OPTS in file
{{hadoop-common-project/hadoop-common/src/main/bin/hadoop}}.
{code}
HADOOP_OPTS=$HADOOP_OPTS $HADOOP_CLIENT_OPTS
{code}
Here $HADOOP_CLIENT_OPTS is set in hadoop-env.sh
({{hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh}})
{code}
export HADOOP_CLIENT_OPTS=-Xmx512m $HADOOP_CLIENT_OPTS
{code}
Currently the final default java command looks like:
{code}java -Xmx1000m -Xmx512m CLASS_NAME ARGUMENTS{code}
And if users also specify the -Xmx in the $HADOOP_CLIENT_OPTS, there will be
three -Xmx configurations.
The hadoop setup tutorial only discusses hadoop-env.sh, and it looks that
users should not make any change in hadoop-config.sh.
We should let hadoop smart to choose the right one before launching the java
command, instead of leaving for jvm to make the decision.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-9109) Support remote shell comands other than ssh in startup scripts
[ https://issues.apache.org/jira/browse/HADOOP-9109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9109: - Resolution: Duplicate Status: Resolved (was: Patch Available) Closing this as a dupe of HADOOP-9902. Support remote shell comands other than ssh in startup scripts -- Key: HADOOP-9109 URL: https://issues.apache.org/jira/browse/HADOOP-9109 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Albert Chu Priority: Minor Attachments: HADOOP-9109-tag-release-1.1.0.patch, HADOOP-9109.patch Some organizations have remote shell launching mechanisms other than ssh. It would be nice to be able to use these other remote shell commands without hand modifying the startup scripts. I've attached a locally developed patch for this against Hadoop 1.1.0. It modifies conf/hadoop-env.sh, bin/slaves.sh, and bin/hadoop-daemon.sh and supports a HADOOP_SSH_CMD environment variable to allow users to set whatever remote shell command they want to use. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102154#comment-14102154 ] Allen Wittenauer commented on HADOOP-9902: -- Jenkins appears to be pretty horked. Patch clearly applies, there are no tests associated with the shell code, and previous versions applied with no issues so I'm just going to commit -16. Thanks all! Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Resolution: Fixed Status: Resolved (was: Patch Available) Commit to trunk svn rev 1618847. Closing. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10530) Make hadoop trunk build on Java7+ only
[
https://issues.apache.org/jira/browse/HADOOP-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102185#comment-14102185
]
Allen Wittenauer commented on HADOOP-10530:
---
At this point, shouldn't trunk really be jdk 1.8?
Make hadoop trunk build on Java7+ only
--
Key: HADOOP-10530
URL: https://issues.apache.org/jira/browse/HADOOP-10530
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 3.0.0, 2.4.0
Environment: Java 1.7+
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: HADOOP-10530-001.patch, HADOOP-10530-002.patch
As discussed on hadoop-common, hadoop 3 is envisaged to be Java7+ *only*
-this JIRA covers switching the build for this
# maven enforcer plugin to set Java version = {{[1.7)}}
# compiler to set language to java 1.7
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10978) HADOOP_IDENT_STRING is overriden in hadoop-env.sh
[
https://issues.apache.org/jira/browse/HADOOP-10978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102196#comment-14102196
]
Allen Wittenauer commented on HADOOP-10978:
---
This problem has already been fixed in trunk as part of HADOOP-9902.
HADOOP_IDENT_STRING is overriden in hadoop-env.sh
-
Key: HADOOP-10978
URL: https://issues.apache.org/jira/browse/HADOOP-10978
Project: Hadoop Common
Issue Type: Bug
Reporter: Mathias Herberts
Priority: Minor
hadoop-env.sh forces HADOOP_IDENT_STRING to $USER possibly overriding a
previously set value.
Instead hadoop-env.sh should set HADOOP_IDENT_STRING to:
export HADOOP_IDENT_STRING=${HADOOP_IDENT_STRING:-$USER}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10908) [post-HADOOP-9902] Cluster Node Setup needs updating
[ https://issues.apache.org/jira/browse/HADOOP-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10908: -- Summary: [post-HADOOP-9902] Cluster Node Setup needs updating (was: Cluster Node Setup needs updating post-HADOOP-9902) [post-HADOOP-9902] Cluster Node Setup needs updating Key: HADOOP-10908 URL: https://issues.apache.org/jira/browse/HADOOP-10908 Project: Hadoop Common Issue Type: Bug Reporter: Allen Wittenauer Assignee: Allen Wittenauer A lot of the instructions in the cluster node setup are not good practices post-9902. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10787) [post-HADOOP-9902] Rename/remove DEFAULT_LIBEXEC_DIR from the shell scripts
[ https://issues.apache.org/jira/browse/HADOOP-10787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10787: -- Summary: [post-HADOOP-9902] Rename/remove DEFAULT_LIBEXEC_DIR from the shell scripts (was: Rename DEFAULT_LIBEXEC_DIR from the shell scripts) [post-HADOOP-9902] Rename/remove DEFAULT_LIBEXEC_DIR from the shell scripts --- Key: HADOOP-10787 URL: https://issues.apache.org/jira/browse/HADOOP-10787 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer DEFAULT_LIBEXEC_DIR pollutes the shell name space. It should be renamed to HADOOP_DEFAULT_LIBEXEC_DIR. Unfortunately, this touches every single shell script. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10879) [post-HADOOP-9902] Rename *-env.sh in the tree to *-env.sh.example
[ https://issues.apache.org/jira/browse/HADOOP-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10879: -- Summary: [post-HADOOP-9902] Rename *-env.sh in the tree to *-env.sh.example (was: Rename *-env.sh in the tree to *-env.sh.example) [post-HADOOP-9902] Rename *-env.sh in the tree to *-env.sh.example -- Key: HADOOP-10879 URL: https://issues.apache.org/jira/browse/HADOOP-10879 Project: Hadoop Common Issue Type: Improvement Reporter: Allen Wittenauer With HADOOP-9902 in place, we don't have to ship *-env.sh called as such and only provide examples. This goes a long way with being able to upgrade the binaries in place since we would no longer overwrite those files upon extraction. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10854) [post-HADOOP-9902] unit tests for the shell scripts
[ https://issues.apache.org/jira/browse/HADOOP-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10854: -- Summary: [post-HADOOP-9902] unit tests for the shell scripts (was: unit tests for the shell scripts) [post-HADOOP-9902] unit tests for the shell scripts --- Key: HADOOP-10854 URL: https://issues.apache.org/jira/browse/HADOOP-10854 Project: Hadoop Common Issue Type: Test Reporter: Allen Wittenauer With HADOOP-9902 moving a lot of functionality to functions, we should build some unit tests for them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10854) [post-HADOOP-9902] unit tests for the shell scripts
[ https://issues.apache.org/jira/browse/HADOOP-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10854: -- Description: With HADOOP-9902 moving a lot of the core functionality to functions, we should build some unit tests for them. (was: With HADOOP-9902 moving a lot of functionality to functions, we should build some unit tests for them.) [post-HADOOP-9902] unit tests for the shell scripts --- Key: HADOOP-10854 URL: https://issues.apache.org/jira/browse/HADOOP-10854 Project: Hadoop Common Issue Type: Test Reporter: Allen Wittenauer With HADOOP-9902 moving a lot of the core functionality to functions, we should build some unit tests for them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10979) [post-HADOOP-9902] Auto-entries in hadoop_usage
Allen Wittenauer created HADOOP-10979: - Summary: [post-HADOOP-9902] Auto-entries in hadoop_usage Key: HADOOP-10979 URL: https://issues.apache.org/jira/browse/HADOOP-10979 Project: Hadoop Common Issue Type: Improvement Reporter: Allen Wittenauer Priority: Minor It would make adding common options to hadoop_usage output easier if some entries were auto-populated. This is similar to what happens in FsShell and other parts of the Java code. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10979) [post-HADOOP-9902] Auto-entries in hadoop_usage
[ https://issues.apache.org/jira/browse/HADOOP-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102209#comment-14102209 ] Allen Wittenauer commented on HADOOP-10979: --- In particular: * --config * --daemon options are standardized across all of the subsystems * help * version? [post-HADOOP-9902] Auto-entries in hadoop_usage --- Key: HADOOP-10979 URL: https://issues.apache.org/jira/browse/HADOOP-10979 Project: Hadoop Common Issue Type: Improvement Reporter: Allen Wittenauer Priority: Minor It would make adding common options to hadoop_usage output easier if some entries were auto-populated. This is similar to what happens in FsShell and other parts of the Java code. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10901) [post-HADOOP-9902] provide un-camelCased versions of shell commands
[ https://issues.apache.org/jira/browse/HADOOP-10901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10901: -- Summary: [post-HADOOP-9902] provide un-camelCased versions of shell commands (was: provide un-camelCased versions of shell commands) [post-HADOOP-9902] provide un-camelCased versions of shell commands --- Key: HADOOP-10901 URL: https://issues.apache.org/jira/browse/HADOOP-10901 Project: Hadoop Common Issue Type: Improvement Reporter: Allen Wittenauer There is a heavy disposition to do camelCase subcommands because it reflects what is in the Java code. However, it is very counter to the shell. We should replace the case options to accept both the camelCase and the fully lowercase options. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10978) HADOOP_IDENT_STRING is overriden in hadoop-env.sh
[
https://issues.apache.org/jira/browse/HADOOP-10978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-10978:
--
Affects Version/s: 2.6.0
HADOOP_IDENT_STRING is overriden in hadoop-env.sh
-
Key: HADOOP-10978
URL: https://issues.apache.org/jira/browse/HADOOP-10978
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Mathias Herberts
Priority: Minor
hadoop-env.sh forces HADOOP_IDENT_STRING to $USER possibly overriding a
previously set value.
Instead hadoop-env.sh should set HADOOP_IDENT_STRING to:
export HADOOP_IDENT_STRING=${HADOOP_IDENT_STRING:-$USER}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10950) rework heap management vars
[ https://issues.apache.org/jira/browse/HADOOP-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10950: -- Hadoop Flags: Incompatible change rework heap management vars - Key: HADOOP-10950 URL: https://issues.apache.org/jira/browse/HADOOP-10950 Project: Hadoop Common Issue Type: Improvement Reporter: Allen Wittenauer Post-HADOOP-9902, we need to rework how heap is configured for small footprint machines, deprecate some options, introduce new ones for greater flexibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10893) isolated classloader on the client side
[
https://issues.apache.org/jira/browse/HADOOP-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14103194#comment-14103194
]
Allen Wittenauer commented on HADOOP-10893:
---
{code}
+# If HADOOP_USE_CLIENT_CLASSLOADER is set, user classes and their dependencies
+# as defined by HADOOP_CLASSPATH and the jar as the hadoop jar argument are
+# loaded by a separate classloader. It should not be mixed with
+# HADOOP_USER_CLASSPATH_FIRST. If it is set, HADOOP_USER_CLASSPATH_FIRST is
+# ignored. Can be defined by doing
+# export HADOOP_USE_CLIENT_CLASSLOADER=true
+
+# HADOOP_CLIENT_CLASSLOADER_SYSTEM_CLASSES overrides the default definition of
+# system classes for the client classloader. The system classes are a
+# comma-separated list of classes that should be loaded from the system
+# classpath, not the user-supplied JARs, when HADOOP_USE_CLIENT_CLASSLOADER is
+# enabled. Names ending in '.' (period) are treated as package names, and names
+# starting with a '-' are treated as negative matches.
+
{code}
I'm not a fan of this wall of text sitting in hadoop-env.sh. Ideally, this
should really be in documentation with a very light description here; that
second paragraph seems too much. Additionally, burying the variable in the
middle of the description is confusing. It should be the last thing in the
section so that it is clear that's what one needs to change. In other words,
follow the pattern established elsewhere.
The change to hadoop_add_to_classpath_userpath looks fine, based upon my
understanding of what this patch is doing.
isolated classloader on the client side
---
Key: HADOOP-10893
URL: https://issues.apache.org/jira/browse/HADOOP-10893
Project: Hadoop Common
Issue Type: New Feature
Components: util
Affects Versions: 2.4.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: HADOOP-10893.patch, HADOOP-10893.patch,
HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch,
HADOOP-10893.patch, classloader-test.tar.gz
We have the job classloader on the mapreduce tasks that run on the cluster.
It has a benefit of being able to isolate class space for user code and avoid
version clashes.
Although it occurs less often, version clashes do occur on the client JVM. It
would be good to introduce an isolated classloader on the client side as well
to address this. A natural point to introduce this may be through RunJar, as
that's how most of hadoop jobs are run.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10893) isolated classloader on the client side
[ https://issues.apache.org/jira/browse/HADOOP-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14103196#comment-14103196 ] Allen Wittenauer commented on HADOOP-10893: --- OK, I see the mistake I made. There is no example export line for HADOOP_CLIENT_CLASSLOADER_SYSTEM_CLASSES so I thought it was still describing the first one. So yeah, add that instead. ;) isolated classloader on the client side --- Key: HADOOP-10893 URL: https://issues.apache.org/jira/browse/HADOOP-10893 Project: Hadoop Common Issue Type: New Feature Components: util Affects Versions: 2.4.0 Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, classloader-test.tar.gz We have the job classloader on the mapreduce tasks that run on the cluster. It has a benefit of being able to isolate class space for user code and avoid version clashes. Although it occurs less often, version clashes do occur on the client JVM. It would be good to introduce an isolated classloader on the client side as well to address this. A natural point to introduce this may be through RunJar, as that's how most of hadoop jobs are run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10893) isolated classloader on the client side
[ https://issues.apache.org/jira/browse/HADOOP-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14103886#comment-14103886 ] Allen Wittenauer commented on HADOOP-10893: --- Yeah, the export line examples make a huge difference. Thanks. isolated classloader on the client side --- Key: HADOOP-10893 URL: https://issues.apache.org/jira/browse/HADOOP-10893 Project: Hadoop Common Issue Type: New Feature Components: util Affects Versions: 2.4.0 Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: HADOOP-10893-branch-2.patch, HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, classloader-test.tar.gz We have the job classloader on the mapreduce tasks that run on the cluster. It has a benefit of being able to isolate class space for user code and avoid version clashes. Although it occurs less often, version clashes do occur on the client JVM. It would be good to introduce an isolated classloader on the client side as well to address this. A natural point to introduce this may be through RunJar, as that's how most of hadoop jobs are run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10979) [post-HADOOP-9902] Auto-entries in hadoop_usage
[ https://issues.apache.org/jira/browse/HADOOP-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10979: -- Component/s: scripts [post-HADOOP-9902] Auto-entries in hadoop_usage --- Key: HADOOP-10979 URL: https://issues.apache.org/jira/browse/HADOOP-10979 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Priority: Minor It would make adding common options to hadoop_usage output easier if some entries were auto-populated. This is similar to what happens in FsShell and other parts of the Java code. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10950) rework heap management vars
[ https://issues.apache.org/jira/browse/HADOOP-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10950: -- Component/s: scripts rework heap management vars - Key: HADOOP-10950 URL: https://issues.apache.org/jira/browse/HADOOP-10950 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Labels: scripts Post-HADOOP-9902, we need to rework how heap is configured for small footprint machines, deprecate some options, introduce new ones for greater flexibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10879) [post-HADOOP-9902] Rename *-env.sh in the tree to *-env.sh.example
[ https://issues.apache.org/jira/browse/HADOOP-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10879: -- Component/s: scripts [post-HADOOP-9902] Rename *-env.sh in the tree to *-env.sh.example -- Key: HADOOP-10879 URL: https://issues.apache.org/jira/browse/HADOOP-10879 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Labels: scripts With HADOOP-9902 in place, we don't have to ship *-env.sh called as such and only provide examples. This goes a long way with being able to upgrade the binaries in place since we would no longer overwrite those files upon extraction. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10901) [post-HADOOP-9902] provide un-camelCased versions of shell commands
[ https://issues.apache.org/jira/browse/HADOOP-10901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10901: -- Component/s: scripts [post-HADOOP-9902] provide un-camelCased versions of shell commands --- Key: HADOOP-10901 URL: https://issues.apache.org/jira/browse/HADOOP-10901 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Labels: scripts There is a heavy disposition to do camelCase subcommands because it reflects what is in the Java code. However, it is very counter to the shell. We should replace the case options to accept both the camelCase and the fully lowercase options. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10854) [post-HADOOP-9902] unit tests for the shell scripts
[ https://issues.apache.org/jira/browse/HADOOP-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10854: -- Component/s: scripts [post-HADOOP-9902] unit tests for the shell scripts --- Key: HADOOP-10854 URL: https://issues.apache.org/jira/browse/HADOOP-10854 Project: Hadoop Common Issue Type: Test Components: scripts Reporter: Allen Wittenauer Labels: scripts With HADOOP-9902 moving a lot of the core functionality to functions, we should build some unit tests for them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10950) rework heap management vars
[ https://issues.apache.org/jira/browse/HADOOP-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10950: -- Release Note: The default heap sizes have been removed. This will allow for the JVM to use auto-tuning based upon the memory size of the host. To re-enable the old default, configure HADOOP_HEAPSIZE_MAX=1g in hadoop-env.sh. rework heap management vars - Key: HADOOP-10950 URL: https://issues.apache.org/jira/browse/HADOOP-10950 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Labels: scripts Post-HADOOP-9902, we need to rework how heap is configured for small footprint machines, deprecate some options, introduce new ones for greater flexibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10950) rework heap management vars
[ https://issues.apache.org/jira/browse/HADOOP-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10950: -- Release Note: * The default heap sizes have been removed and the HADOOP_HEAPSIZE variable has been deprecated (It will still be honored if set, but expect it to go away in the future). This will allow for the JVM to use auto-tuning based upon the memory size of the host. To re-enable the old default, configure HADOOP_HEAPSIZE_MAX=1g in hadoop-env.sh. * All heap size variables now support units. If the variable is only a number, the size is assumed to be in megabytes. was: * The default heap sizes have been removed and the HADOOP_HEAPSIZE variable has been deprecated (It will still be honored if set, but expect it to go away in the future). This will allow for the JVM to use auto-tuning based upon the memory size of the host. To re-enable the old default, configure HADOOP_HEAPSIZE_MAX=1g in hadoop-env.sh. * All heap size variables now support units. If the variable is only a number, the size assumed to be in megabytes. rework heap management vars - Key: HADOOP-10950 URL: https://issues.apache.org/jira/browse/HADOOP-10950 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Labels: scripts Post-HADOOP-9902, we need to rework how heap is configured for small footprint machines, deprecate some options, introduce new ones for greater flexibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10950) rework heap management vars
[ https://issues.apache.org/jira/browse/HADOOP-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10950: -- Release Note: * The default heap sizes have been removed and the HADOOP_HEAPSIZE variable has been deprecated (It will still be honored if set, but expect it to go away in the future). This will allow for the JVM to use auto-tuning based upon the memory size of the host. To re-enable the old default, configure HADOOP_HEAPSIZE_MAX=1g in hadoop-env.sh. * All heap size variables now support units. If the variable is only a number, the size assumed to be in megabytes. was:The default heap sizes have been removed. This will allow for the JVM to use auto-tuning based upon the memory size of the host. To re-enable the old default, configure HADOOP_HEAPSIZE_MAX=1g in hadoop-env.sh. rework heap management vars - Key: HADOOP-10950 URL: https://issues.apache.org/jira/browse/HADOOP-10950 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Labels: scripts Post-HADOOP-9902, we need to rework how heap is configured for small footprint machines, deprecate some options, introduce new ones for greater flexibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10788) [post-HADOOP-9902] Rewrite httpfs, kms, sls, and other stragglers
[ https://issues.apache.org/jira/browse/HADOOP-10788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104082#comment-14104082 ] Allen Wittenauer commented on HADOOP-10788: --- I started to take a look at this one. This is going to be a bit more work than expected. Most of the work for httpfs and kms is being done by the tomcat script. So this means that there is not only excessive verbosity coming from our code but also from tomcat. Additionally, the hadoop_daemon_* functions were built from the perspective that hadoop-common would be launching Java. This is clearly not the case with these two daemons. In order to kill two birds with one stone, I'm thinking the best approach would be to make Yet Another Set of non-java daemon functions that could be used generically. We'd wrap all of the tomcat stuff such that instead of going to the screen it'd go to .out instead. [post-HADOOP-9902] Rewrite httpfs, kms, sls, and other stragglers - Key: HADOOP-10788 URL: https://issues.apache.org/jira/browse/HADOOP-10788 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Labels: scripts There are some stragglers not targeted by HADOOP-9902. These should also get rewritten to use the new hadoop-functions.sh framework. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10788) [post-HADOOP-9902] Rewrite httpfs, kms, sls, and other stragglers
[
https://issues.apache.org/jira/browse/HADOOP-10788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104091#comment-14104091
]
Allen Wittenauer commented on HADOOP-10788:
---
Another approach might be:
{code}
HADOOP_USE_JAVA=${HADOOP_USE_JAVA:-true}
...
if [[ ${HADOOP_USE_JAVA} eq true ]]; then
exec ${JAVA} blah
else
exec blah
fi
{code}
That might be less code overall. Hmm.
[post-HADOOP-9902] Rewrite httpfs, kms, sls, and other stragglers
-
Key: HADOOP-10788
URL: https://issues.apache.org/jira/browse/HADOOP-10788
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Labels: scripts
There are some stragglers not targeted by HADOOP-9902. These should also get
rewritten to use the new hadoop-functions.sh framework.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10787) [post-HADOOP-9902] Rename/remove non-HADOOP_*, non-YARN_* from the shell scripts
[ https://issues.apache.org/jira/browse/HADOOP-10787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10787: -- Description: We should make an effort to clean up the shell env var name space by removing unsafe variables. See comments for list. (was: We should make an effort to clean up the shell env var name space) [post-HADOOP-9902] Rename/remove non-HADOOP_*, non-YARN_* from the shell scripts Key: HADOOP-10787 URL: https://issues.apache.org/jira/browse/HADOOP-10787 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Labels: scripts We should make an effort to clean up the shell env var name space by removing unsafe variables. See comments for list. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10787) [post-HADOOP-9902] Rename/remove non-HADOOP_*, non-YARN_* from the shell scripts
[ https://issues.apache.org/jira/browse/HADOOP-10787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10787: -- Summary: [post-HADOOP-9902] Rename/remove non-HADOOP_*, non-YARN_* from the shell scripts (was: [post-HADOOP-9902] Rename/remove DEFAULT_LIBEXEC_DIR from the shell scripts) [post-HADOOP-9902] Rename/remove non-HADOOP_*, non-YARN_* from the shell scripts Key: HADOOP-10787 URL: https://issues.apache.org/jira/browse/HADOOP-10787 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Labels: scripts DEFAULT_LIBEXEC_DIR pollutes the shell name space. It should be renamed to HADOOP_DEFAULT_LIBEXEC_DIR. Unfortunately, this touches every single shell script. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10787) [post-HADOOP-9902] Rename/remove non-HADOOP_*, non-YARN_* from the shell scripts
[ https://issues.apache.org/jira/browse/HADOOP-10787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10787: -- Description: We should make an effort to clean up the shell env var name space (was: DEFAULT_LIBEXEC_DIR pollutes the shell name space. It should be renamed to HADOOP_DEFAULT_LIBEXEC_DIR. Unfortunately, this touches every single shell script.) [post-HADOOP-9902] Rename/remove non-HADOOP_*, non-YARN_* from the shell scripts Key: HADOOP-10787 URL: https://issues.apache.org/jira/browse/HADOOP-10787 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Labels: scripts We should make an effort to clean up the shell env var name space -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10787) [post-HADOOP-9902] Rename/remove non-HADOOP_*, non-YARN_* from the shell scripts
[ https://issues.apache.org/jira/browse/HADOOP-10787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104099#comment-14104099 ] Allen Wittenauer commented on HADOOP-10787: --- List: * DEFAULT_LIBEXEC_DIR * TOOL_PATH * JAVA_HEAP_MAX (handled by HADOOP-10950) * SLAVE_NAMES? (just need to be made local?) * SLAVE_FILE? (just need to be made local?) Not to be removed: * HADOOP_* * YARN_* * MAPRED_* Non-Hadoop vars intentionally set/used: * JAVA_HOME * PDSH_SSH_ARGS_APPEND * LD_LIBRARY_PATH (for people who don't know how to use ld -R...) * JAVA_LIBRARY_PATH (ditto) * MALLOC_ARENA_MAX * EUID * JAVA (although maybe this should get changed to HADOOP_JAVA?) * JSVC_HOME * CLASSPATH * BASH_VERSINFO NOTE: list currently ignores the craziness that is httpfs, kms, etc. [post-HADOOP-9902] Rename/remove non-HADOOP_*, non-YARN_* from the shell scripts Key: HADOOP-10787 URL: https://issues.apache.org/jira/browse/HADOOP-10787 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Labels: scripts We should make an effort to clean up the shell env var name space by removing unsafe variables. See comments for list. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10787) [post-HADOOP-9902] Rename/remove non-HADOOP_*, etc from the shell scripts
[ https://issues.apache.org/jira/browse/HADOOP-10787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10787: -- Summary: [post-HADOOP-9902] Rename/remove non-HADOOP_*, etc from the shell scripts (was: [post-HADOOP-9902] Rename/remove non-HADOOP_*, non-YARN_* from the shell scripts) [post-HADOOP-9902] Rename/remove non-HADOOP_*, etc from the shell scripts - Key: HADOOP-10787 URL: https://issues.apache.org/jira/browse/HADOOP-10787 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Labels: scripts We should make an effort to clean up the shell env var name space by removing unsafe variables. See comments for list. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10950) rework heap management vars
[ https://issues.apache.org/jira/browse/HADOOP-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10950: -- Attachment: HADOOP-10950.patch rework heap management vars - Key: HADOOP-10950 URL: https://issues.apache.org/jira/browse/HADOOP-10950 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Labels: scripts Attachments: HADOOP-10950.patch Post-HADOOP-9902, we need to rework how heap is configured for small footprint machines, deprecate some options, introduce new ones for greater flexibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10950) rework heap management vars
[ https://issues.apache.org/jira/browse/HADOOP-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10950: -- Assignee: Allen Wittenauer Status: Patch Available (was: Open) rework heap management vars - Key: HADOOP-10950 URL: https://issues.apache.org/jira/browse/HADOOP-10950 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts Attachments: HADOOP-10950.patch Post-HADOOP-9902, we need to rework how heap is configured for small footprint machines, deprecate some options, introduce new ones for greater flexibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10950) rework heap management vars
[ https://issues.apache.org/jira/browse/HADOOP-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14104357#comment-14104357 ] Allen Wittenauer commented on HADOOP-10950: --- This patch also fixes some documentation issues from HADOOP-9902 wrt heap sizes. rework heap management vars - Key: HADOOP-10950 URL: https://issues.apache.org/jira/browse/HADOOP-10950 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts Attachments: HADOOP-10950.patch Post-HADOOP-9902, we need to rework how heap is configured for small footprint machines, deprecate some options, introduce new ones for greater flexibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10950) rework heap management vars
[ https://issues.apache.org/jira/browse/HADOOP-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10950: -- Release Note: * The default heap sizes have been removed and the HADOOP_HEAPSIZE variable has been deprecated (It will still be honored if set, but expect it to go away in the future). This will allow for the JVM to use auto-tuning based upon the memory size of the host. To re-enable the old default, configure HADOOP_HEAPSIZE_MAX=1g in hadoop-env.sh. * All global and daemon-specific heap size variables now support units. If the variable is only a number, the size is assumed to be in megabytes. was: * The default heap sizes have been removed and the HADOOP_HEAPSIZE variable has been deprecated (It will still be honored if set, but expect it to go away in the future). This will allow for the JVM to use auto-tuning based upon the memory size of the host. To re-enable the old default, configure HADOOP_HEAPSIZE_MAX=1g in hadoop-env.sh. * All heap size variables now support units. If the variable is only a number, the size is assumed to be in megabytes. rework heap management vars - Key: HADOOP-10950 URL: https://issues.apache.org/jira/browse/HADOOP-10950 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts Attachments: HADOOP-10950.patch Post-HADOOP-9902, we need to rework how heap is configured for small footprint machines, deprecate some options, introduce new ones for greater flexibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10946) Fix a bunch of typos in log messages
[ https://issues.apache.org/jira/browse/HADOOP-10946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10946: -- Status: Open (was: Patch Available) Fix a bunch of typos in log messages Key: HADOOP-10946 URL: https://issues.apache.org/jira/browse/HADOOP-10946 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.1 Reporter: Ray Chiang Priority: Trivial Labels: newbie Attachments: HADOOP-10946-04.patch, HADOOP10946-01.patch, HADOOP10946-02.patch, HADOOP10946-03.patch There are a bunch of typos in various log messages. These need cleaning up. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10946) Fix a bunch of typos in log messages
[ https://issues.apache.org/jira/browse/HADOOP-10946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10946: -- Status: Patch Available (was: Open) Fix a bunch of typos in log messages Key: HADOOP-10946 URL: https://issues.apache.org/jira/browse/HADOOP-10946 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.1 Reporter: Ray Chiang Priority: Trivial Labels: newbie Attachments: HADOOP-10946-04.patch, HADOOP10946-01.patch, HADOOP10946-02.patch, HADOOP10946-03.patch There are a bunch of typos in various log messages. These need cleaning up. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10946) Fix a bunch of typos in log messages
[ https://issues.apache.org/jira/browse/HADOOP-10946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10946: -- Assignee: Ray Chiang Fix a bunch of typos in log messages Key: HADOOP-10946 URL: https://issues.apache.org/jira/browse/HADOOP-10946 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.1 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Trivial Labels: newbie Attachments: HADOOP-10946-04.patch, HADOOP10946-01.patch, HADOOP10946-02.patch, HADOOP10946-03.patch There are a bunch of typos in various log messages. These need cleaning up. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-8896) Javadoc points to Wrong Reader and Writer classes in SequenceFile
[
https://issues.apache.org/jira/browse/HADOOP-8896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-8896:
-
Assignee: Ray Chiang
Javadoc points to Wrong Reader and Writer classes in SequenceFile
-
Key: HADOOP-8896
URL: https://issues.apache.org/jira/browse/HADOOP-8896
Project: Hadoop Common
Issue Type: Improvement
Components: documentation, io
Affects Versions: 2.0.1-alpha
Reporter: Timothy Mann
Assignee: Ray Chiang
Priority: Trivial
Labels: sequence-file
Attachments: HADOOP-8896-02.patch, HADOOP-8896-03.patch,
HADOOP8896-01.patch
Original Estimate: 1m
Remaining Estimate: 1m
Line 56 of org.apache.hadoop.io.SequenceFile refers to {@link Writer}, {@link
Reader} in the javadoc comment describing the class SequenceFile. When the
javadoc is built Reader and Writer link to java.io.Reader and java.io.Writer,
respectively. However, they should instead refer to {@link
SequenceFile.Reader} and {@link SequenceFile.Writer}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105351#comment-14105351
]
Allen Wittenauer commented on HADOOP-9902:
--
bq. yarn command usage info seems broken. e.g. yarn application command
earlier was printing command usage info. Now, it's throwing exception.
Looks like I missed this command line stack manipulation for ApplicationCLI:
{code}
elif [ $COMMAND = application ] ||
[ $COMMAND = applicationattempt ] ||
[ $COMMAND = container ]; then
CLASS=org.apache.hadoop.yarn.client.cli.ApplicationCLI
YARN_OPTS=$YARN_OPTS $YARN_CLIENT_OPTS
set -- $COMMAND $@
{code}
... probably because it is a very oddball thing to do. I'll file a JIRA for
that.
bq. Starting/stopping yarn daemon doesn't print anything any more. Earlier it
was printing something like starting resource manager.., which I think is
useful.
Putting it inside yarn-daemon.sh or anywhere else breaks the init.d script
experience for ops teams. So, if anything, this should get changed in
yarn-daemons.sh and make it more of an analog to hadoop-daemons.sh.
bq. we should open separate jiras in YARN/MR to track YARN/MR side changes, so
that it draws enough attention in YARN/MR community as well.
It's an interesting data point to note that the follow-up JIRAs for this one to
fix bugs, add a few more features, etc, are surprisingly light with watchers,
if they even have any. Probably hints as to another reason why this part of
the code base never gets fixes. It was decided early on (see above) to do this
as one big JIRA. That was still, IMO, the correct decision based upon history
and the current state.
While this was a sweeping change across all of the subprojects, all of these
individual communities should be paying attention to what is happening in
common due to the dependency structure.
Shell script rewrite
Key: HADOOP-9902
URL: https://issues.apache.org/jira/browse/HADOOP-9902
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: releasenotes
Fix For: 3.0.0
Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch,
HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch,
HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch,
HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch,
HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch,
HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt,
hadoop-9902-1.patch, more-info.txt
Umbrella JIRA for shell script rewrite. See more-info.txt for more details.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105355#comment-14105355 ] Allen Wittenauer commented on HADOOP-9902: -- I've file YARN-2436 and YARN-2437 (under the new script component I added yesterday...) for those two issues. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105355#comment-14105355 ] Allen Wittenauer edited comment on HADOOP-9902 at 8/21/14 1:10 PM: --- I've filed YARN-2436 and YARN-2437 (under the new script component I added yesterday...) for those two issues. was (Author: aw): I've file YARN-2436 and YARN-2437 (under the new script component I added yesterday...) for those two issues. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10991) 'hadoop namenode -format' fails if user hadoop homedir is not under /home
[
https://issues.apache.org/jira/browse/HADOOP-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-10991.
---
Resolution: Cannot Reproduce
This is either:
a) permissions issue in /home/hadoop
b) misconfiguration in hdfs-site.xml
I've run namenode format in many many directories, including /home and not
/home.
It should be noted that changing HADOOP_PREFIX has zero influence on where the
NN fsimage gets created.
Closing as cannot reproduce.
'hadoop namenode -format' fails if user hadoop homedir is not under /home
-
Key: HADOOP-10991
URL: https://issues.apache.org/jira/browse/HADOOP-10991
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.5.0
Environment: CentOS 6.5
Reporter: Yaniv Kaul
Priority: Minor
Labels: namenode
Since my homedir is on shared NFS, I couldn't create a user for hadoop on
/home. Therefore, I've used:
useradd hadoop --home /hadoop
which worked well. I've adjusted HADOOP_HOME and friends to match correctly.
Running hdfs namenode -format failed:
{noformat}
14/08/21 13:57:35 INFO namenode.NNConf: XAttrs enabled? true
14/08/21 13:57:35 INFO namenode.NNConf: Maximum size of an xattr: 16384
14/08/21 13:57:35 INFO namenode.FSImage: Allocated new BlockPoolId:
BP-1696511243-10.103.234.197-1408618655940
14/08/21 13:57:35 WARN namenode.NameNode: Encountered exception during
format:
java.io.IOException: Cannot create directory
/home/hadoop/hadoopdata/hdfs/namenode/current
at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:334)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:546)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:567)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:148)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:926)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1354)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1473)
14/08/21 13:57:35 FATAL namenode.NameNode: Exception in namenode join
java.io.IOException: Cannot create directory
/home/hadoop/hadoopdata/hdfs/namenode/current
at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:334)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:546)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:567)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:148)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:926)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1354)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1473)
14/08/21 13:57:35 INFO util.ExitUtil: Exiting with status 1
14/08/21 13:57:35 INFO namenode.NameNode: SHUTDOWN_MSG:
/
SHUTDOWN_MSG: Shutting down NameNode at
lgdrm432.xiodrm.lab.emc.com/10.103.234.197
{noformat}
The error seems to be clear:
{noformat}
java.io.IOException: Cannot create directory
/home/hadoop/hadoopdata/hdfs/namenode/current
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10950) rework heap management vars
[ https://issues.apache.org/jira/browse/HADOOP-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10950: -- Attachment: HADOOP-10950-01.patch -01: fix some typos, better desc of _MAX and _MIN. Rebased. rework heap management vars - Key: HADOOP-10950 URL: https://issues.apache.org/jira/browse/HADOOP-10950 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts Attachments: HADOOP-10950-01.patch, HADOOP-10950.patch Post-HADOOP-9902, we need to rework how heap is configured for small footprint machines, deprecate some options, introduce new ones for greater flexibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10899) Hadoop CommandsManual.vm documentation gives deprecated information
[ https://issues.apache.org/jira/browse/HADOOP-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-10899. --- Resolution: Fixed yes, probably time to close this one. Thanks for working on them! Hadoop CommandsManual.vm documentation gives deprecated information --- Key: HADOOP-10899 URL: https://issues.apache.org/jira/browse/HADOOP-10899 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Allen Wittenauer Assignee: Akira AJISAKA Labels: newbie This is a rollup of several old JIRAs. The CommandsManual lists very old information about running HDFS and MapReduce subcommands from the 'hadoop' shell CLI. These are deprecated and should be removed. If necessary, the commands should be added to the relevant subproject's documentation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10946) Fix a bunch of typos in log messages
[ https://issues.apache.org/jira/browse/HADOOP-10946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10946: -- Affects Version/s: 3.0.0 Fix a bunch of typos in log messages Key: HADOOP-10946 URL: https://issues.apache.org/jira/browse/HADOOP-10946 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.4.1 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Trivial Labels: newbie Attachments: HADOOP-10946-04.patch, HADOOP10946-01.patch, HADOOP10946-02.patch, HADOOP10946-03.patch There are a bunch of typos in various log messages. These need cleaning up. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10946) Fix a bunch of typos in log messages
[ https://issues.apache.org/jira/browse/HADOOP-10946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105486#comment-14105486 ] Allen Wittenauer commented on HADOOP-10946: --- I've triggered a manual jenkins test. (I think.) So weird. Fix a bunch of typos in log messages Key: HADOOP-10946 URL: https://issues.apache.org/jira/browse/HADOOP-10946 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, 2.4.1 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Trivial Labels: newbie Attachments: HADOOP-10946-04.patch, HADOOP10946-01.patch, HADOOP10946-02.patch, HADOOP10946-03.patch There are a bunch of typos in various log messages. These need cleaning up. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10893) isolated classloader on the client side
[ https://issues.apache.org/jira/browse/HADOOP-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105819#comment-14105819 ] Allen Wittenauer commented on HADOOP-10893: --- tl;dr: [~sjlee0]'s changes here are probably the correct ones. From a pure patch perspective, it does look weird. But from a stylistic perspective as a part of a total work (namely, hadoop 2.x), the changes and lack of documentation in hadoop-env.sh, etc, to branch-2 make a lot of sense. One of the key points of HADOOP-9902 was to highlight to end users what things they could set. Hiding that in hadoop-config.sh, which users are never directed to documentation-wise, didn't really work. So I pulled those out and popped them into hadoop-env.sh, which users definitely see. This change just got caught in the crossfire. isolated classloader on the client side --- Key: HADOOP-10893 URL: https://issues.apache.org/jira/browse/HADOOP-10893 Project: Hadoop Common Issue Type: New Feature Components: util Affects Versions: 2.4.0 Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: HADOOP-10893-branch-2.patch, HADOOP-10893-branch-2.patch, HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, HADOOP-10893.patch, classloader-test.tar.gz We have the job classloader on the mapreduce tasks that run on the cluster. It has a benefit of being able to isolate class space for user code and avoid version clashes. Although it occurs less often, version clashes do occur on the client JVM. It would be good to introduce an isolated classloader on the client side as well to address this. A natural point to introduce this may be through RunJar, as that's how most of hadoop jobs are run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105924#comment-14105924
]
Allen Wittenauer commented on HADOOP-9902:
--
bq. Found one more problem that RM and NM daemon logs are now inside *.out
file, instead of *.log file.
As mentioned in the release notes, YARN did a bunch of heinous stuff when it
came to log4j settings, counter to the rest of Hadoop, and to much frustration
with ops teams. This has been made consistent, so there is good chance you
were relying upon that behavior. It could be any number of things: NM and RM
_OPT settings, dependence upon the nodemanger/log4j.settings file or
resourcemanager/log4j.settings file, yarn-env.sh settings, etc. You can always
do 'bash -x yarn --daemon start resourcemanager'. The out file should contain
the java command line.
With the shipping *-env.sh files, you should see something similar to:
{code}
java -Dproc_resourcemanager -Djava.net.preferIPv4Stack=true
-Djava.security.krb5.realm= -Djava.security.krb5.kdc=
-Djava.security.krb5.conf=
-Dyarn.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
-Dyarn.log.file=hadoop-aw-resourcemanager-aw-mbp-work.local.log
-Dyarn.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT
-Dyarn.root.logger=INFO,RFA -Xmx1g
-Dhadoop.log.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT/logs
-Dhadoop.log.file=hadoop-aw-resourcemanager-aw-mbp-work.local.log
-Dhadoop.home.dir=/Users/aw/HADOOP/hadoop-3.0.0-SNAPSHOT -Dhadoop.id.str=aw
-Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml
-Dhadoop.security.logger=INFO,NullAppender
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
{code}
bq. Also yarn daemon log file name earlier was starting with yarn, now it's
changed to hadoop, not sure if this is an intentional change.
Intentional. This is to make YARN consistent with the rest of the system.
(Noticing a theme?)
Shell script rewrite
Key: HADOOP-9902
URL: https://issues.apache.org/jira/browse/HADOOP-9902
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: releasenotes
Fix For: 3.0.0
Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch,
HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch,
HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch,
HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch,
HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch,
HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt,
hadoop-9902-1.patch, more-info.txt
Umbrella JIRA for shell script rewrite. See more-info.txt for more details.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105960#comment-14105960
]
Allen Wittenauer commented on HADOOP-9902:
--
... and, just to answer the question before it gets asked... ;)
Want to override what the RM uses for logging? Just put this in the
yarn-env.sh:
{code}
export YARN_RESOURCEMANAGER_OPTS=-Dhadoop.root.logger=INFO,DRFA
-Dyarn.root.logger=INFO,DRFA
{code}
The rest of the system will fill in the blanks.
Shell script rewrite
Key: HADOOP-9902
URL: https://issues.apache.org/jira/browse/HADOOP-9902
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: releasenotes
Fix For: 3.0.0
Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch,
HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch,
HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch,
HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch,
HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch,
HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt,
hadoop-9902-1.patch, more-info.txt
Umbrella JIRA for shell script rewrite. See more-info.txt for more details.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10950) rework heap management vars
[ https://issues.apache.org/jira/browse/HADOOP-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10950: -- Status: Patch Available (was: Open) rework heap management vars - Key: HADOOP-10950 URL: https://issues.apache.org/jira/browse/HADOOP-10950 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts Attachments: HADOOP-10950-01.patch, HADOOP-10950.patch Post-HADOOP-9902, we need to rework how heap is configured for small footprint machines, deprecate some options, introduce new ones for greater flexibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10950) rework heap management vars
[ https://issues.apache.org/jira/browse/HADOOP-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10950: -- Status: Open (was: Patch Available) rework heap management vars - Key: HADOOP-10950 URL: https://issues.apache.org/jira/browse/HADOOP-10950 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts Attachments: HADOOP-10950-01.patch, HADOOP-10950.patch Post-HADOOP-9902, we need to rework how heap is configured for small footprint machines, deprecate some options, introduce new ones for greater flexibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10950) rework heap management vars
[ https://issues.apache.org/jira/browse/HADOOP-10950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105971#comment-14105971 ] Allen Wittenauer commented on HADOOP-10950: --- Jenkins tests, of course, are fairly meaningless for this patch. rework heap management vars - Key: HADOOP-10950 URL: https://issues.apache.org/jira/browse/HADOOP-10950 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts Attachments: HADOOP-10950-01.patch, HADOOP-10950.patch Post-HADOOP-9902, we need to rework how heap is configured for small footprint machines, deprecate some options, introduce new ones for greater flexibility. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10993) Dump java command line to *.out file
Allen Wittenauer created HADOOP-10993: - Summary: Dump java command line to *.out file Key: HADOOP-10993 URL: https://issues.apache.org/jira/browse/HADOOP-10993 Project: Hadoop Common Issue Type: Improvement Reporter: Allen Wittenauer Priority: Minor It might be a nice enhancement to print the contents of the java command line to the out file during daemon startup to help with debugging. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-8896) Javadoc points to Wrong Reader and Writer classes in SequenceFile
[
https://issues.apache.org/jira/browse/HADOOP-8896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-8896:
-
Resolution: Fixed
Fix Version/s: 2.6.0
3.0.0
Status: Resolved (was: Patch Available)
+1. Committing to trunk and branch-2.
Thanks!
Javadoc points to Wrong Reader and Writer classes in SequenceFile
-
Key: HADOOP-8896
URL: https://issues.apache.org/jira/browse/HADOOP-8896
Project: Hadoop Common
Issue Type: Improvement
Components: documentation, io
Affects Versions: 2.0.1-alpha
Reporter: Timothy Mann
Assignee: Ray Chiang
Priority: Trivial
Labels: sequence-file
Fix For: 3.0.0, 2.6.0
Attachments: HADOOP-8896-02.patch, HADOOP-8896-03.patch,
HADOOP8896-01.patch
Original Estimate: 1m
Remaining Estimate: 1m
Line 56 of org.apache.hadoop.io.SequenceFile refers to {@link Writer}, {@link
Reader} in the javadoc comment describing the class SequenceFile. When the
javadoc is built Reader and Writer link to java.io.Reader and java.io.Writer,
respectively. However, they should instead refer to {@link
SequenceFile.Reader} and {@link SequenceFile.Writer}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106294#comment-14106294 ] Allen Wittenauer commented on HADOOP-9902: -- Hmm it's basically looking for a working libexec. You need a dir with hadoop-functions.sh, hadoop-config.sh, and hdfs-config.sh (for hdfs). So you could construct that manually and point HADOOP_LIBEXEC_DIR to it and I think all would work. But I definitely see a bug and/or feature here, depending upon ones world view. ;) I've filed a separate jira to cover this case (HADOOP-10996 ), as it's not an insignificant amount of work. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10996) run hdfs, yarn, mapred, etc from build tree
Allen Wittenauer created HADOOP-10996: - Summary: run hdfs, yarn, mapred, etc from build tree Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer There is a developer use case for running the shell scripts from the build tree. What would it take to make it work? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10996) run hdfs, yarn, mapred, etc from build tree
[
https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106305#comment-14106305
]
Allen Wittenauer commented on HADOOP-10996:
---
Given this:
{code}
$ export HADOOP_COMMON_HOME=$(pwd)/$(ls -d
hadoop-common-project/hadoop-common/target/hadoop-common-*/)
$ export HADOOP_HDFS_HOME=$(pwd)/$(ls -d
hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*/)
$ export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$PATH
$ hdfs
ERROR: Unable to exec
(path)target/hadoop-hdfs-3.0.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh.
{code}
How do we make hdfs work properly?
First, what is happening?
The code tries to find where to look for hdfs-config.sh is located. It does
this by looking for ../libexec, where it finds it. It now makes the (false)
assumption that this must be the libexec dir. So it now tries to fire up
hadoop-config.sh and hadoop-functions.sh which fail.
There are a couple of different ways to solve this:
* Look to see if HADOOP_COMMON_HOME is defined and look for
hadoop-config.sh/hadoop-functions.sh is there as well.
* Throw caution to the wind and see if this stuff is in our current path.
* Do the full gamut of checks for HADOOP_HDFS_HOME, etc, for hdfs-config.sh +
the stuff above.
One sticking point is what happens if hadoop-layout.sh redefines the directory
structure? The code is sort of in a catch-22.
run hdfs, yarn, mapred, etc from build tree
---
Key: HADOOP-10996
URL: https://issues.apache.org/jira/browse/HADOOP-10996
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
There is a developer use case for running the shell scripts from the build
tree. What would it take to make it work?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HADOOP-10996) run hdfs, yarn, mapred, etc from build tree
[
https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106305#comment-14106305
]
Allen Wittenauer edited comment on HADOOP-10996 at 8/22/14 1:30 AM:
Given this:
{code}
$ export HADOOP_COMMON_HOME=$(pwd)/$(ls -d
hadoop-common-project/hadoop-common/target/hadoop-common-*/)
$ export HADOOP_HDFS_HOME=$(pwd)/$(ls -d
hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*/)
$ export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$PATH
$ hdfs
ERROR: Unable to exec
(path)target/hadoop-hdfs-3.0.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh.
{code}
How do we make hdfs work properly?
First, what is happening?
The code tries to find where to look for hdfs-config.sh is located. It does
this by looking for ../libexec, where it finds it. It now makes the (false)
assumption that this must be the one, true libexec dir. So it now tries to
fire up hadoop-config.sh and hadoop-functions.sh which fail.
There are a couple of different ways to solve this:
* Look to see if HADOOP_COMMON_HOME is defined and look for
hadoop-config.sh/hadoop-functions.sh is there as well.
* Throw caution to the wind and see if this stuff is in our current path.
* Do the full gamut of checks for HADOOP_HDFS_HOME, etc, for hdfs-config.sh +
the stuff above.
One sticking point is what happens if hadoop-layout.sh redefines the directory
structure? The code is sort of in a catch-22.
was (Author: aw):
Given this:
{code}
$ export HADOOP_COMMON_HOME=$(pwd)/$(ls -d
hadoop-common-project/hadoop-common/target/hadoop-common-*/)
$ export HADOOP_HDFS_HOME=$(pwd)/$(ls -d
hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*/)
$ export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$PATH
$ hdfs
ERROR: Unable to exec
(path)target/hadoop-hdfs-3.0.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh.
{code}
How do we make hdfs work properly?
First, what is happening?
The code tries to find where to look for hdfs-config.sh is located. It does
this by looking for ../libexec, where it finds it. It now makes the (false)
assumption that this must be the libexec dir. So it now tries to fire up
hadoop-config.sh and hadoop-functions.sh which fail.
There are a couple of different ways to solve this:
* Look to see if HADOOP_COMMON_HOME is defined and look for
hadoop-config.sh/hadoop-functions.sh is there as well.
* Throw caution to the wind and see if this stuff is in our current path.
* Do the full gamut of checks for HADOOP_HDFS_HOME, etc, for hdfs-config.sh +
the stuff above.
One sticking point is what happens if hadoop-layout.sh redefines the directory
structure? The code is sort of in a catch-22.
run hdfs, yarn, mapred, etc from build tree
---
Key: HADOOP-10996
URL: https://issues.apache.org/jira/browse/HADOOP-10996
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
There is a developer use case for running the shell scripts from the build
tree. What would it take to make it work?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HADOOP-10996) run hdfs, yarn, mapred, etc from build tree
[
https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106305#comment-14106305
]
Allen Wittenauer edited comment on HADOOP-10996 at 8/22/14 1:31 AM:
(Source:
https://issues.apache.org/jira/secure/EditComment!default.jspa?id=12665400commentId=14106278)
Given this:
{code}
$ export HADOOP_COMMON_HOME=$(pwd)/$(ls -d
hadoop-common-project/hadoop-common/target/hadoop-common-*/)
$ export HADOOP_HDFS_HOME=$(pwd)/$(ls -d
hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*/)
$ export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$PATH
$ hdfs
ERROR: Unable to exec
(path)target/hadoop-hdfs-3.0.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh.
{code}
How do we make hdfs work properly?
First, what is happening?
The code tries to find where to look for hdfs-config.sh is located. It does
this by looking for ../libexec, where it finds it. It now makes the (false)
assumption that this must be the one, true libexec dir. So it now tries to
fire up hadoop-config.sh and hadoop-functions.sh which fail.
There are a couple of different ways to solve this:
* Look to see if HADOOP_COMMON_HOME is defined and look for
hadoop-config.sh/hadoop-functions.sh is there as well.
* Throw caution to the wind and see if this stuff is in our current path.
* Do the full gamut of checks for HADOOP_HDFS_HOME, etc, for hdfs-config.sh +
the stuff above.
One sticking point is what happens if hadoop-layout.sh redefines the directory
structure? The code is sort of in a catch-22.
was (Author: aw):
Given this:
{code}
$ export HADOOP_COMMON_HOME=$(pwd)/$(ls -d
hadoop-common-project/hadoop-common/target/hadoop-common-*/)
$ export HADOOP_HDFS_HOME=$(pwd)/$(ls -d
hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*/)
$ export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$PATH
$ hdfs
ERROR: Unable to exec
(path)target/hadoop-hdfs-3.0.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh.
{code}
How do we make hdfs work properly?
First, what is happening?
The code tries to find where to look for hdfs-config.sh is located. It does
this by looking for ../libexec, where it finds it. It now makes the (false)
assumption that this must be the one, true libexec dir. So it now tries to
fire up hadoop-config.sh and hadoop-functions.sh which fail.
There are a couple of different ways to solve this:
* Look to see if HADOOP_COMMON_HOME is defined and look for
hadoop-config.sh/hadoop-functions.sh is there as well.
* Throw caution to the wind and see if this stuff is in our current path.
* Do the full gamut of checks for HADOOP_HDFS_HOME, etc, for hdfs-config.sh +
the stuff above.
One sticking point is what happens if hadoop-layout.sh redefines the directory
structure? The code is sort of in a catch-22.
run hdfs, yarn, mapred, etc from build tree
---
Key: HADOOP-10996
URL: https://issues.apache.org/jira/browse/HADOOP-10996
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
There is a developer use case for running the shell scripts from the build
tree. What would it take to make it work?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10996) run hdfs, yarn, mapred, etc from target tree
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10996: -- Summary: run hdfs, yarn, mapred, etc from target tree (was: run hdfs, yarn, mapred, etc from build tree) run hdfs, yarn, mapred, etc from target tree Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer There is a developer use case for running the shell scripts from the build tree. What would it take to make it work? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HADOOP-10996) run hdfs, yarn, mapred, etc from target tree
[
https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106305#comment-14106305
]
Allen Wittenauer edited comment on HADOOP-10996 at 8/22/14 1:33 AM:
(Source:
https://issues.apache.org/jira/secure/EditComment!default.jspa?id=12665400commentId=14106278)
Given this:
{code}
$ export HADOOP_COMMON_HOME=$(pwd)/$(ls -d
hadoop-common-project/hadoop-common/target/hadoop-common-*/)
$ export HADOOP_HDFS_HOME=$(pwd)/$(ls -d
hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*/)
$ export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$PATH
$ hdfs
ERROR: Unable to exec
(path)target/hadoop-hdfs-3.0.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh.
{code}
How do we make hdfs work properly?
First, what is happening?
The code tries to find where to look for hdfs-config.sh is located. It does
this by looking for ../libexec, where it finds it. It now makes the (false)
assumption that this must be the one, true libexec dir. So it now tries to
fire up hadoop-config.sh and hadoop-functions.sh which fail.
There are a couple of different ways to solve this:
* Look to see if HADOOP_COMMON_HOME is defined and look for
hadoop-config.sh/hadoop-functions.sh is there as well.
* Throw caution to the wind and see if this stuff is in our current path.
* Recalculate HADOOP_LIBEXEC_DIR in hadoop-config.sh might work too, since
clearly hdfs found it.
* Do the full gamut of checks for HADOOP_HDFS_HOME, etc, for hdfs-config.sh +
some of the stuff above.
One sticking point is what happens if hadoop-layout.sh redefines the directory
structure? The code is sort of in a catch-22.
was (Author: aw):
(Source:
https://issues.apache.org/jira/secure/EditComment!default.jspa?id=12665400commentId=14106278)
Given this:
{code}
$ export HADOOP_COMMON_HOME=$(pwd)/$(ls -d
hadoop-common-project/hadoop-common/target/hadoop-common-*/)
$ export HADOOP_HDFS_HOME=$(pwd)/$(ls -d
hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-*/)
$ export PATH=$HADOOP_COMMON_HOME/bin:$HADOOP_HDFS_HOME/bin:$PATH
$ hdfs
ERROR: Unable to exec
(path)target/hadoop-hdfs-3.0.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh.
{code}
How do we make hdfs work properly?
First, what is happening?
The code tries to find where to look for hdfs-config.sh is located. It does
this by looking for ../libexec, where it finds it. It now makes the (false)
assumption that this must be the one, true libexec dir. So it now tries to
fire up hadoop-config.sh and hadoop-functions.sh which fail.
There are a couple of different ways to solve this:
* Look to see if HADOOP_COMMON_HOME is defined and look for
hadoop-config.sh/hadoop-functions.sh is there as well.
* Throw caution to the wind and see if this stuff is in our current path.
* Do the full gamut of checks for HADOOP_HDFS_HOME, etc, for hdfs-config.sh +
the stuff above.
One sticking point is what happens if hadoop-layout.sh redefines the directory
structure? The code is sort of in a catch-22.
run hdfs, yarn, mapred, etc from target tree
Key: HADOOP-10996
URL: https://issues.apache.org/jira/browse/HADOOP-10996
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
There is a developer use case for running the shell scripts from the build
tree. What would it take to make it work?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HADOOP-10996) run hdfs, yarn, mapred, etc from target tree
[
https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106321#comment-14106321
]
Allen Wittenauer edited comment on HADOOP-10996 at 8/22/14 1:41 AM:
OK, hdfs-config.sh does to the right thing (although it could be argued the
order should be reversed):
{code}
if [ -e ${HADOOP_LIBEXEC_DIR}/hadoop-config.sh ]; then
. ${HADOOP_LIBEXEC_DIR}/hadoop-config.sh
elif [ -e ${HADOOP_COMMON_HOME}/libexec/hadoop-config.sh ]; then
. ${HADOOP_COMMON_HOME}/libexec/hadoop-config.sh
elif [ -e ${HADOOP_HOME}/libexec/hadoop-config.sh ]; then
. ${HADOOP_HOME}/libexec/hadoop-config.sh
else
echo ERROR: Hadoop common not found. 21
exit 1
fi
{code}
So it's really hadoop-config.sh that's broken here:
{code}
# get our functions defined for usage later
if [[ -f ${HADOOP_LIBEXEC_DIR}/hadoop-functions.sh ]]; then
. ${HADOOP_LIBEXEC_DIR}/hadoop-functions.sh
else
echo ERROR: Unable to exec ${HADOOP_LIBEXEC_DIR}/hadoop-functions.sh. 12
exit 1
fi
# allow overrides of the above and pre-defines of the below
if [[ -f ${HADOOP_LIBEXEC_DIR}/hadoop-layout.sh ]]; then
. ${HADOOP_LIBEXEC_DIR}/hadoop-layout.sh
fi
{code}
This is going to be a relatively easy fix, I think. We just need to add checks
for HADOOP_COMMON_HOME prior to using HADOOP_LIBEXEC_DIR.
was (Author: aw):
OK, hdfs-config.sh does to the right thing:
{code}
if [ -e ${HADOOP_LIBEXEC_DIR}/hadoop-config.sh ]; then
. ${HADOOP_LIBEXEC_DIR}/hadoop-config.sh
elif [ -e ${HADOOP_COMMON_HOME}/libexec/hadoop-config.sh ]; then
. ${HADOOP_COMMON_HOME}/libexec/hadoop-config.sh
elif [ -e ${HADOOP_HOME}/libexec/hadoop-config.sh ]; then
. ${HADOOP_HOME}/libexec/hadoop-config.sh
else
echo ERROR: Hadoop common not found. 21
exit 1
fi
{code}
So it's really hadoop-config.sh that's broken here:
{code}
# get our functions defined for usage later
if [[ -f ${HADOOP_LIBEXEC_DIR}/hadoop-functions.sh ]]; then
. ${HADOOP_LIBEXEC_DIR}/hadoop-functions.sh
else
echo ERROR: Unable to exec ${HADOOP_LIBEXEC_DIR}/hadoop-functions.sh. 12
exit 1
fi
# allow overrides of the above and pre-defines of the below
if [[ -f ${HADOOP_LIBEXEC_DIR}/hadoop-layout.sh ]]; then
. ${HADOOP_LIBEXEC_DIR}/hadoop-layout.sh
fi
{code}
This is going to be a relatively easy fix, I think. We just need to add checks
for HADOOP_COMMON_HOME prior to using HADOOP_LIBEXEC_DIR.
run hdfs, yarn, mapred, etc from target tree
Key: HADOOP-10996
URL: https://issues.apache.org/jira/browse/HADOOP-10996
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
There is a developer use case for running the shell scripts from the build
tree. What would it take to make it work?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10996) run hdfs, yarn, mapred, etc from target tree
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10996: -- Attachment: HADOOP-10996.patch Patch that fixes hadoop-config.sh to use HADOOP_COMMON_HOME/libexec if it can't find it in HADOOP_LIBEXEC_DIR as well as fixes two bugs in HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME definitions when they aren't defined. run hdfs, yarn, mapred, etc from target tree Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Attachments: HADOOP-10996.patch There is a developer use case for running the shell scripts from the build tree. What would it take to make it work? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10996) run hdfs, yarn, mapred, etc from target tree
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106350#comment-14106350 ] Allen Wittenauer commented on HADOOP-10996: --- FWIW, I opted to reverse the order because I remembered why I did it in the other code as well: in NORMAL operating modes, HADOOP_LIBEXEC_DIR is the correct place. run hdfs, yarn, mapred, etc from target tree Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Attachments: HADOOP-10996.patch There is a developer use case for running the shell scripts from the build tree. What would it take to make it work? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10996) run hdfs, yarn, mapred, etc from target tree
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10996: -- Attachment: HADOOP-10996-01.patch -01: Wait... wait... wait... We should NOT be using HADOOP_HOME for anything! So let's fix that too. run hdfs, yarn, mapred, etc from target tree Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Attachments: HADOOP-10996-01.patch, HADOOP-10996.patch There is a developer use case for running the shell scripts from the build tree. What would it take to make it work? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HADOOP-10996) run hdfs, yarn, mapred, etc from target tree
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer reassigned HADOOP-10996: - Assignee: Allen Wittenauer run hdfs, yarn, mapred, etc from target tree Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-10996-01.patch, HADOOP-10996.patch There is a developer use case for running the shell scripts from the build tree. What would it take to make it work? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10996) run hdfs, yarn, mapred, etc from target tree
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10996: -- Status: Patch Available (was: Open) run hdfs, yarn, mapred, etc from target tree Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Attachments: HADOOP-10996-01.patch, HADOOP-10996.patch There is a developer use case for running the shell scripts from the build tree. What would it take to make it work? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-10998) Bash tab completion support for newer versions of hadoop
[
https://issues.apache.org/jira/browse/HADOOP-10998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106828#comment-14106828
]
Allen Wittenauer commented on HADOOP-10998:
---
It'd probably be better if you took at a look at and updated HADOOP-6310, as it
has some other fixes in it as well I think.
Bash tab completion support for newer versions of hadoop
Key: HADOOP-10998
URL: https://issues.apache.org/jira/browse/HADOOP-10998
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Environment: Hadoop 2.0.0-cdh4.2.1
Reporter: Jim Hester
Priority: Trivial
Attachments: HADOOP-10998.patch
The included version of bash completion has not been updated in almost two
years and does not work correctly with more recent versions of Hadoop.
The command substitutions in the script were using backticks, which were
interpolating variables before they were executed and breaking the script.
In particular the awk commands were breaking due to this interpolation.
The attached patch changes the backticks to {{$(command)}} style for command
substitution and updates the script to work properly with newer versions of
hadoop.
I tested the script using {{zsh 5.0.5}}, {{zsh 4.3.10]}, {{bash-4.1.2}} and
{{bash-3.2}}. Hadoop versions tested were cloudera version
{{2.0.0-cdh4.2.1}} and briefly Apache {{2.5.0}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10996) [post-HADOOP-9902] Stop the violence in the *_HOME
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10996: -- Summary: [post-HADOOP-9902] Stop the violence in the *_HOME (was: run hdfs, yarn, mapred, etc from target tree) [post-HADOOP-9902] Stop the violence in the *_HOME -- Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-10996-01.patch, HADOOP-10996.patch There is a developer use case for running the shell scripts from the build tree. What would it take to make it work? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10996) [post-HADOOP-9902] Stop the violence in the *_HOME
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10996: -- Description: (Updated from original description) There are various places where the various HOME directories are missing or mis-defined. was:There are various places where the various HOME directories are missing or mis-defined. [post-HADOOP-9902] Stop the violence in the *_HOME -- Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-10996-01.patch, HADOOP-10996.patch (Updated from original description) There are various places where the various HOME directories are missing or mis-defined. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10996) [post-HADOOP-9902] Stop violence in the *_HOME
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10996: -- Summary: [post-HADOOP-9902] Stop violence in the *_HOME (was: [post-HADOOP-9902] Stop the violence in the *_HOME) [post-HADOOP-9902] Stop violence in the *_HOME -- Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-10996-01.patch, HADOOP-10996.patch (Updated from original description) There are various places where the various HOME directories are missing or mis-defined. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10996) [post-HADOOP-9902] Stop the violence in the *_HOME
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10996: -- Description: There are various places where the various HOME directories are missing or mis-defined. (was: There is a developer use case for running the shell scripts from the build tree. What would it take to make it work?) [post-HADOOP-9902] Stop the violence in the *_HOME -- Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-10996-01.patch, HADOOP-10996.patch There are various places where the various HOME directories are missing or mis-defined. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10998) Bash tab completion support for newer versions of hadoop
[
https://issues.apache.org/jira/browse/HADOOP-10998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-10998:
--
Resolution: Fixed
Fix Version/s: 2.6.0
3.0.0
Status: Resolved (was: Patch Available)
+1, I'll commit this to branch-2 and trunk. Thanks!
For future generations:
The bash tab completion code still has a lot of problems. Ideally, we'd fix
them all with one patch that would be a rewrite of the current code. (In fact,
I wouldn't recommend people actually use this feature at all.) But at least
this patch moves the code forward in a positive way. There is still a lot of
work to be done.
Bash tab completion support for newer versions of hadoop
Key: HADOOP-10998
URL: https://issues.apache.org/jira/browse/HADOOP-10998
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Environment: Hadoop 2.0.0-cdh4.2.1
Reporter: Jim Hester
Priority: Trivial
Fix For: 3.0.0, 2.6.0
Attachments: HADOOP-10998.patch
The included version of bash completion has not been updated in almost two
years and does not work correctly with more recent versions of Hadoop.
The command substitutions in the script were using backticks, which were
interpolating variables before they were executed and breaking the script.
In particular the awk commands were breaking due to this interpolation.
The attached patch changes the backticks to {{$(command)}} style for command
substitution and updates the script to work properly with newer versions of
hadoop.
I tested the script using {{zsh 5.0.5}}, {{zsh 4.3.10]}, {{bash-4.1.2}} and
{{bash-3.2}}. Hadoop versions tested were cloudera version
{{2.0.0-cdh4.2.1}} and briefly Apache {{2.5.0}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10998) Fix bash tab completion code to work
[
https://issues.apache.org/jira/browse/HADOOP-10998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-10998:
--
Summary: Fix bash tab completion code to work (was: Bash tab completion
support for newer versions of hadoop)
Fix bash tab completion code to work
Key: HADOOP-10998
URL: https://issues.apache.org/jira/browse/HADOOP-10998
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Environment: Hadoop 2.0.0-cdh4.2.1
Reporter: Jim Hester
Priority: Trivial
Fix For: 3.0.0, 2.6.0
Attachments: HADOOP-10998.patch
The included version of bash completion has not been updated in almost two
years and does not work correctly with more recent versions of Hadoop.
The command substitutions in the script were using backticks, which were
interpolating variables before they were executed and breaking the script.
In particular the awk commands were breaking due to this interpolation.
The attached patch changes the backticks to {{$(command)}} style for command
substitution and updates the script to work properly with newer versions of
hadoop.
I tested the script using {{zsh 5.0.5}}, {{zsh 4.3.10]}, {{bash-4.1.2}} and
{{bash-3.2}}. Hadoop versions tested were cloudera version
{{2.0.0-cdh4.2.1}} and briefly Apache {{2.5.0}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10998) Fix bash tab completion code to work
[
https://issues.apache.org/jira/browse/HADOOP-10998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-10998:
--
Assignee: Jim Hester
Fix bash tab completion code to work
Key: HADOOP-10998
URL: https://issues.apache.org/jira/browse/HADOOP-10998
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Environment: Hadoop 2.0.0-cdh4.2.1
Reporter: Jim Hester
Assignee: Jim Hester
Priority: Trivial
Fix For: 3.0.0, 2.6.0
Attachments: HADOOP-10998.patch
The included version of bash completion has not been updated in almost two
years and does not work correctly with more recent versions of Hadoop.
The command substitutions in the script were using backticks, which were
interpolating variables before they were executed and breaking the script.
In particular the awk commands were breaking due to this interpolation.
The attached patch changes the backticks to {{$(command)}} style for command
substitution and updates the script to work properly with newer versions of
hadoop.
I tested the script using {{zsh 5.0.5}}, {{zsh 4.3.10]}, {{bash-4.1.2}} and
{{bash-3.2}}. Hadoop versions tested were cloudera version
{{2.0.0-cdh4.2.1}} and briefly Apache {{2.5.0}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-6310) bash completion doesn't quite work.
[ https://issues.apache.org/jira/browse/HADOOP-6310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6310. -- Resolution: Duplicate Closing this as a dupe of HADOOP-10998. See commit comment there. bash completion doesn't quite work. --- Key: HADOOP-6310 URL: https://issues.apache.org/jira/browse/HADOOP-6310 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.18.3 Reporter: Paul Huff Assignee: Sean Mackrory Priority: Trivial Attachments: HADOOP-6310.patch, HADOOP-6310.patch.1 The bash completion script in src/contrib/bash-tab-completion/hadoop.sh doesn't quite work the way you'd expect it to against 18.3 (and I assume anything afterwards, since the author claimed compatibility with 16-20). It doesn't complete things like you'd expect against HDFS, and it doesn't have job-id completion. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10996) [post-HADOOP-9902] Stop violence in the *_HOME
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10996: -- Status: Open (was: Patch Available) Cancelling patch -01. After working with it, I've found some edge and not-so-edge cases that either: a) are made worse (usage of *_HOME should be viewed as exceptions to _PREFIX, not as an all or nothing scenario) b) aren't covered (etc/hadoop/*-site.xml comes from *_HOME) [post-HADOOP-9902] Stop violence in the *_HOME -- Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-10996-01.patch, HADOOP-10996.patch (Updated from original description) There are various places where the various HOME directories are missing or mis-defined. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HADOOP-10996) [post-HADOOP-9902] Stop violence in the *_HOME
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109162#comment-14109162 ] Allen Wittenauer edited comment on HADOOP-10996 at 8/25/14 2:57 PM: Cancelling patch -01. After working with it, I've found some edge and not-so-edge cases that either: a) are made worse (e.g., usage of *_HOME should be viewed as exceptions to _PREFIX, not as an all or nothing scenario) b) aren't covered (e.g., etc/hadoop/*-site.xml comes from *_HOME) was (Author: aw): Cancelling patch -01. After working with it, I've found some edge and not-so-edge cases that either: a) are made worse (usage of *_HOME should be viewed as exceptions to _PREFIX, not as an all or nothing scenario) b) aren't covered (etc/hadoop/*-site.xml comes from *_HOME) [post-HADOOP-9902] Stop violence in the *_HOME -- Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-10996-01.patch, HADOOP-10996.patch (Updated from original description) There are various places where the various HOME directories are missing or mis-defined. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-11002) shell escapes are incompatible with previous releases
Allen Wittenauer created HADOOP-11002:
-
Summary: shell escapes are incompatible with previous releases
Key: HADOOP-11002
URL: https://issues.apache.org/jira/browse/HADOOP-11002
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Reporter: Allen Wittenauer
Post-HADOOP-9902, the following in namenode opts doesn't work without being
escaped:
{code}
-XX:HeapDumpPath=./java_pid_pid.hprof
{code}
This is a bit of surprising behavior to the users. The breakage is directly
result of the code that fixes spaces in directories. Since it is much more
likely to hit weird metacharacters in shell than have directories with spaces,
that part of HADOOP-9902 needs to get replaced.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14109673#comment-14109673 ] Allen Wittenauer commented on HADOOP-9902: -- FYI: HADOOP-11002 - shell escapes are incompatible with previous releases Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-11002) shell escapes are incompatible with previous releases
[
https://issues.apache.org/jira/browse/HADOOP-11002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-11002:
--
Description:
Post-HADOOP-9902, the following in xyz_OPTS doesn't work without being escaped:
{code}
-XX:HeapDumpPath=./java_pid_pid.hprof
{code}
This is a bit of surprising behavior to the users. The breakage is directly
result of the code that fixes spaces in directories. Since it is much more
likely to hit weird metacharacters in shell than have directories with spaces,
that part of HADOOP-9902 needs to get replaced.
was:
Post-HADOOP-9902, the following in namenode opts doesn't work without being
escaped:
{code}
-XX:HeapDumpPath=./java_pid_pid.hprof
{code}
This is a bit of surprising behavior to the users. The breakage is directly
result of the code that fixes spaces in directories. Since it is much more
likely to hit weird metacharacters in shell than have directories with spaces,
that part of HADOOP-9902 needs to get replaced.
shell escapes are incompatible with previous releases
-
Key: HADOOP-11002
URL: https://issues.apache.org/jira/browse/HADOOP-11002
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Reporter: Allen Wittenauer
Post-HADOOP-9902, the following in xyz_OPTS doesn't work without being
escaped:
{code}
-XX:HeapDumpPath=./java_pid_pid.hprof
{code}
This is a bit of surprising behavior to the users. The breakage is directly
result of the code that fixes spaces in directories. Since it is much more
likely to hit weird metacharacters in shell than have directories with
spaces, that part of HADOOP-9902 needs to get replaced.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-11002) shell escapes are incompatible with previous releases
[
https://issues.apache.org/jira/browse/HADOOP-11002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14110201#comment-14110201
]
Allen Wittenauer commented on HADOOP-11002:
---
After playing with this for most of today, I've come to the conclusion that
we're going to have to drop support for directories with spaces. There's
simply no, 100% reliable way to make them work with a global var like
HADOOP_OPTS without doing some really bizarre tricks that break other things.
Support for metacharacters like is significantly more important.
shell escapes are incompatible with previous releases
-
Key: HADOOP-11002
URL: https://issues.apache.org/jira/browse/HADOOP-11002
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Reporter: Allen Wittenauer
Labels: regression
Post-HADOOP-9902, the following in xyz_OPTS doesn't work without being
escaped:
{code}
-XX:HeapDumpPath=./java_pid_pid.hprof
{code}
This is a bit of surprising behavior to the users. The breakage is directly
result of the code that fixes spaces in directories. Since it is much more
likely to hit weird metacharacters in shell than have directories with
spaces, that part of HADOOP-9902 needs to get replaced.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-11002) shell escapes are incompatible with previous releases
[
https://issues.apache.org/jira/browse/HADOOP-11002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-11002:
--
Labels: regression (was: )
shell escapes are incompatible with previous releases
-
Key: HADOOP-11002
URL: https://issues.apache.org/jira/browse/HADOOP-11002
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Reporter: Allen Wittenauer
Labels: regression
Post-HADOOP-9902, the following in xyz_OPTS doesn't work without being
escaped:
{code}
-XX:HeapDumpPath=./java_pid_pid.hprof
{code}
This is a bit of surprising behavior to the users. The breakage is directly
result of the code that fixes spaces in directories. Since it is much more
likely to hit weird metacharacters in shell than have directories with
spaces, that part of HADOOP-9902 needs to get replaced.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
Additionally, pid files are now created when daemons are run in interactive
mode. This will also prevent the accidental starting of two daemons with the
same configuration prior to launching java (i.e., fast fail without having to
wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly alterated, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS_*, YARN_*, and MAPRED_* environment variables act as overrides to
their equivalent HADOOP_* environment variables when 'hdfs', 'yarn', 'mapred',
and related commands are executed. Previously, these were separated out which
meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files via *_OPT.
* Some formerly 'documented' entries in yarn-env.sh have been undocumented as a
simple form of deprecration in order to greatly simplify configuration and
reduce unnecessary duplication. They will still work, but those variables will
likely be removed in a future release.
* Support for various undocumentented YARN log4j.properties files has been
removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented yarn.id.str has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP_SSH_PARALLEL} to prevent memory and network exhaustion.
By default, this is set to 10.
* ${HADOOP_CLIENT_OPTS} support has been added to a few more commands.
* Some subcommands were not listed in the usage.
* Various options on hadoop command lines were supported inconsistently. These
have been unified into hadoop-config.sh. --config is still required to be
first, however.
* ulimit logging for secure daemons no longer assumes /bin/bash but does assume
bash is on the command line path.
* Removed references to some Yahoo! specific paths.
* Removed unused slaves.sh from YARN build tree.
* Many exit states have been changed to reflect reality.
* Shell level errors now go to STDERR. Before, many of them went incorrectly
to STDOUT.
* CDPATH with a period (.) should no longer break the scripts.
* The scripts no longer try to chown directories.
IMPROVEMENTS:
* The *.out files are now appended instead of overwritten to allow for external
log rotation.
* The style and layout of the scripts is much more consistent across
subprojects.
* More of the shell code is now commented.
* Significant amounts of redundant code have been moved
[jira] [Commented] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14110218#comment-14110218 ] Allen Wittenauer commented on HADOOP-9902: -- Given bug fix JIRAs are getting little-to-no traction in watchers, much less reviews, some advice on how to proceed on what are clearly bugs (HADOOP-10996, HADOOP-11002, likely more as they come in) would be appreciated... Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-11002) shell escapes are incompatible with previous releases
[
https://issues.apache.org/jira/browse/HADOOP-11002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-11002:
--
Attachment: HADOOP-11002.patch
This removes support for spaces.
shell escapes are incompatible with previous releases
-
Key: HADOOP-11002
URL: https://issues.apache.org/jira/browse/HADOOP-11002
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Reporter: Allen Wittenauer
Labels: regression
Attachments: HADOOP-11002.patch
Post-HADOOP-9902, the following in xyz_OPTS doesn't work without being
escaped:
{code}
-XX:HeapDumpPath=./java_pid_pid.hprof
{code}
This is a bit of surprising behavior to the users. The breakage is directly
result of the code that fixes spaces in directories. Since it is much more
likely to hit weird metacharacters in shell than have directories with
spaces, that part of HADOOP-9902 needs to get replaced.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HADOOP-10996) [post-HADOOP-9902] Stop violence in the *_HOME
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-10996: -- Attachment: HADOOP-10996-02.patch -02: Changed up the order. Reduces the amount of stat calls needed by checking if some of the *_HOME vars are defined. I started to poke around at enabling *_HOME/etc/hadoop (or whatever), but decided it probably isn't worth it since it will likely lead to unpredictable results. [~andrew.wang], please try this out and see if it fixes your specific issue. Thanks! [post-HADOOP-9902] Stop violence in the *_HOME -- Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-10996-01.patch, HADOOP-10996-02.patch, HADOOP-10996.patch (Updated from original description) There are various places where the various HOME directories are missing or mis-defined. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-11002) shell escapes are incompatible with previous releases
[
https://issues.apache.org/jira/browse/HADOOP-11002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14110255#comment-14110255
]
Allen Wittenauer commented on HADOOP-11002:
---
(It's worth pointing out that directories with spaces are completely broken in
branch-2. HADOOP-9902 made quite a few things work, but without fixing this
issue, not enough to be usable.)
shell escapes are incompatible with previous releases
-
Key: HADOOP-11002
URL: https://issues.apache.org/jira/browse/HADOOP-11002
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Reporter: Allen Wittenauer
Labels: regression
Attachments: HADOOP-11002.patch
Post-HADOOP-9902, the following in xyz_OPTS doesn't work without being
escaped:
{code}
-XX:HeapDumpPath=./java_pid_pid.hprof
{code}
This is a bit of surprising behavior to the users. The breakage is directly
result of the code that fixes spaces in directories. Since it is much more
likely to hit weird metacharacters in shell than have directories with
spaces, that part of HADOOP-9902 needs to get replaced.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-11002) shell escapes are incompatible with previous releases
[
https://issues.apache.org/jira/browse/HADOOP-11002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-11002.
---
Resolution: Fixed
Fix Version/s: 3.0.0
Thanks! I'll commit this as soon as git opens up!
shell escapes are incompatible with previous releases
-
Key: HADOOP-11002
URL: https://issues.apache.org/jira/browse/HADOOP-11002
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Reporter: Allen Wittenauer
Labels: regression
Fix For: 3.0.0
Attachments: HADOOP-11002.patch
Post-HADOOP-9902, the following in xyz_OPTS doesn't work without being
escaped:
{code}
-XX:HeapDumpPath=./java_pid_pid.hprof
{code}
This is a bit of surprising behavior to the users. The breakage is directly
result of the code that fixes spaces in directories. Since it is much more
likely to hit weird metacharacters in shell than have directories with
spaces, that part of HADOOP-9902 needs to get replaced.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HADOOP-11002) shell escapes are incompatible with previous releases
[
https://issues.apache.org/jira/browse/HADOOP-11002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer reassigned HADOOP-11002:
-
Assignee: Allen Wittenauer
shell escapes are incompatible with previous releases
-
Key: HADOOP-11002
URL: https://issues.apache.org/jira/browse/HADOOP-11002
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: regression
Fix For: 3.0.0
Attachments: HADOOP-11002.patch
Post-HADOOP-9902, the following in xyz_OPTS doesn't work without being
escaped:
{code}
-XX:HeapDumpPath=./java_pid_pid.hprof
{code}
This is a bit of surprising behavior to the users. The breakage is directly
result of the code that fixes spaces in directories. Since it is much more
likely to hit weird metacharacters in shell than have directories with
spaces, that part of HADOOP-9902 needs to get replaced.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HADOOP-10996) [post-HADOOP-9902] Stop violence in the *_HOME
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14111776#comment-14111776 ] Allen Wittenauer commented on HADOOP-10996: --- TL;DR: Absolute best bet is to put configs some place and assign HADOOP_CONF_DIR to it so that you have absolute certainty on where Hadoop is pulling settings. Longer story: Currently, if HADOOP_CONF_DIR isn't defined, it uses a bit of twisted logic to locate it: 1. Figure out where HADOOP_PREFIX is at. Is HADOOP_PREFIX defined? If not, then let's assume it's what's called us/... 2. Does HADOOP_PREFIX/conf/hadoop-env.sh exist? OK, then that must be HADOOP_CONF_DIR 3. No? OK, then HADOOP_CONF_DIR must be HADOOP_PREFIX/etc/hadoop. What's fun about this and what you're doing is that HADOOP_CONF_DIR will get defined differently depending upon which bin dir you are using. :D Fine, you say! Let's just treat all *_HOME/etc/hadoop and *_HOME/conf as potentially valid. Now we have a very interesting problem: how do you define HADOOP_CONF_DIR? Other stuff past Hadoop depends upon this being *one* directory. We could pick the first one and then just shove the rest in the classpath and none would be the wiser! Aha! But they would. Which one takes precedence? What happens if there are conflicts? etc, etc. It gets messy very very fast. So... ABORT! ABORT! (BTW, this is pretty much the same logic from branch-2. It could be argued that there should be a check to see if etc/hadoop is 'real' too and abort on it. Here's the fun part: the shell code works perfectly fine if *-env.sh is empty now... the NN will still crash though. That said, if HADOOP-10879 gets finished, this will almost certainly need to get revisited. Probably better to look for core-site.xml, honestly, since all of the sub-projects all depend upon that. In other words, we could run through all of the *_HOME, HADOOP_PREFIX, etc, and use the first core-site.xml we find as the 'real' HADOOP_CONF_DIR.) [post-HADOOP-9902] Stop violence in the *_HOME -- Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-10996-01.patch, HADOOP-10996-02.patch, HADOOP-10996.patch (Updated from original description) There are various places where the various HOME directories are missing or mis-defined. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HADOOP-10996) [post-HADOOP-9902] Stop violence in the *_HOME
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14111776#comment-14111776 ] Allen Wittenauer edited comment on HADOOP-10996 at 8/27/14 3:32 AM: TL;DR: Absolute best bet is to put configs some place and assign HADOOP_CONF_DIR to it so that you have absolute certainty on where Hadoop is pulling settings. Longer story: Currently, if HADOOP_CONF_DIR isn't defined, it uses a bit of interesting logic to locate it: 1. Figure out where HADOOP_PREFIX is at. Is HADOOP_PREFIX defined? If not, then let's assume it's what's called us/... 2. Does HADOOP_PREFIX/conf/hadoop-env.sh exist? OK, then that must be HADOOP_CONF_DIR 3. No? OK, then HADOOP_CONF_DIR must be HADOOP_PREFIX/etc/hadoop. What's fun about this and what you're doing is that HADOOP_CONF_DIR will get defined differently depending upon which bin dir you are using. :D Fine, you say! Let's just treat all _HOME/etc/hadoop and _HOME/conf as potentially valid. Now we have a very interesting problem: how do you define HADOOP_CONF_DIR? Other stuff past Hadoop depends upon this being one directory. We could pick the first one and then just shove the rest in the classpath and none would be the wiser! Aha! But they would. Which one takes precedence? What happens if there are conflicts? etc, etc. It gets messy very very fast. So... ABORT! ABORT! (BTW, this is pretty much the same logic from branch-2. It could be argued that there should be a check to see if etc/hadoop is 'real' too and abort on it. Here's the fun part: the shell code works perfectly fine if -env.sh is empty now... the NN will still crash though. That said, if HADOOP-10879 gets finished, this will almost certainly need to get revisited. Probably better to look for core-site.xml, honestly, since all of the sub-projects all depend upon that. In other words, we could run through all of the _HOME, HADOOP_PREFIX, etc, and use the first core-site.xml we find as the 'real' HADOOP_CONF_DIR.) was (Author: aw): TL;DR: Absolute best bet is to put configs some place and assign HADOOP_CONF_DIR to it so that you have absolute certainty on where Hadoop is pulling settings. Longer story: Currently, if HADOOP_CONF_DIR isn't defined, it uses a bit of twisted logic to locate it: 1. Figure out where HADOOP_PREFIX is at. Is HADOOP_PREFIX defined? If not, then let's assume it's what's called us/... 2. Does HADOOP_PREFIX/conf/hadoop-env.sh exist? OK, then that must be HADOOP_CONF_DIR 3. No? OK, then HADOOP_CONF_DIR must be HADOOP_PREFIX/etc/hadoop. What's fun about this and what you're doing is that HADOOP_CONF_DIR will get defined differently depending upon which bin dir you are using. :D Fine, you say! Let's just treat all *_HOME/etc/hadoop and *_HOME/conf as potentially valid. Now we have a very interesting problem: how do you define HADOOP_CONF_DIR? Other stuff past Hadoop depends upon this being *one* directory. We could pick the first one and then just shove the rest in the classpath and none would be the wiser! Aha! But they would. Which one takes precedence? What happens if there are conflicts? etc, etc. It gets messy very very fast. So... ABORT! ABORT! (BTW, this is pretty much the same logic from branch-2. It could be argued that there should be a check to see if etc/hadoop is 'real' too and abort on it. Here's the fun part: the shell code works perfectly fine if *-env.sh is empty now... the NN will still crash though. That said, if HADOOP-10879 gets finished, this will almost certainly need to get revisited. Probably better to look for core-site.xml, honestly, since all of the sub-projects all depend upon that. In other words, we could run through all of the *_HOME, HADOOP_PREFIX, etc, and use the first core-site.xml we find as the 'real' HADOOP_CONF_DIR.) [post-HADOOP-9902] Stop violence in the *_HOME -- Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-10996-01.patch, HADOOP-10996-02.patch, HADOOP-10996.patch (Updated from original description) There are various places where the various HOME directories are missing or mis-defined. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10996) [post-HADOOP-9902] Stop violence in the *_HOME
[ https://issues.apache.org/jira/browse/HADOOP-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-10996. --- Resolution: Fixed Fix Version/s: 3.0.0 Thanks! I'll commit this as soon as the git repo opens up! [post-HADOOP-9902] Stop violence in the *_HOME -- Key: HADOOP-10996 URL: https://issues.apache.org/jira/browse/HADOOP-10996 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Fix For: 3.0.0 Attachments: HADOOP-10996-01.patch, HADOOP-10996-02.patch, HADOOP-10996.patch (Updated from original description) There are various places where the various HOME directories are missing or mis-defined. -- This message was sent by Atlassian JIRA (v6.2#6252)
