[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP\_IDENT\_STR}. This should allow, with proper
configurations in place, for multiple versions of the same secure daemon to run
on a host. Additionally, pid files are now created when daemons are run in
interactive mode. This will also prevent the accidental starting of two
daemons with the same configuration prior to launching java (i.e., "fast fail"
without having to wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly altered, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS\_\*, YARN\_\*, and MAPRED\_\* environment variables act as overrides
to their equivalent HADOOP\_\* environment variables when 'hdfs', 'yarn',
'mapred', and related commands are executed. Previously, these were separated
out which meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the \*-env.sh files via
\*\_OPT.
* Support for various undocumented YARN log4j.properties files has been removed.
* Support for ${HADOOP\_MASTER} and the related rsync code have been removed.
* The undocumented and unused yarn.id.str Java property has been removed.
* The unused yarn.policy.file Java property has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH\_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP\_\*\_PATH} or
${HADOOP\_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP\_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
NEW FEATURES:
* Daemonization has been moved from *-daemon.sh to the bin commands via the
--daemon option. Simply use --daemon start to start a daemon, --daemon stop to
stop a daemon, and --daemon status to set $? to the daemon's status. The
return code for status is LSB-compatible. For example, 'hdfs --daemon start
namenode'.
* It is now possible to override some of the shell code capabilities to provide
site specific functionality without replacing the shipped versions.
Replacement functions should go into the new hadoop-user-functions.sh file.
* A new option called --buildpaths will attempt to add developer build
directories to the classpath to allow for in source tree testing.
* Operations which trigger ssh connections can now use pdsh if installed.
${HADOOP\_SSH\_OPTS} still gets applied.
* Added distch and jnipath subcommands to the hadoop command.
* Shell scripts now support a --debug option which will report basic
information on the construction of various environment variables, java options,
classpath, etc. to help in configuration debugging.
BUG FIXES:
* ${HADOOP\_CONF\_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP\_CONF\_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP\_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP\_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP\_SSH\_PARALLEL} to prevent memory and network
exhaustion. By default, this is set to 10.
* ${HADOOP\_CLIENT\_OPTS} support has been added to a few more commands.
* Some subcommands were not listed in the usage.
* Various options on
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
Additionally, pid files are now created when daemons are run in interactive
mode. This will also prevent the accidental starting of two daemons with the
same configuration prior to launching java (i.e., "fast fail" without having to
wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly altered, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS_*, YARN_*, and MAPRED_* environment variables act as overrides to
their equivalent HADOOP_* environment variables when 'hdfs', 'yarn', 'mapred',
and related commands are executed. Previously, these were separated out which
meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files via *_OPT.
* Support for various undocumented YARN log4j.properties files has been removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented and unused yarn.id.str Java property has been removed.
* The unused yarn.policy.file Java property has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
NEW FEATURES:
* Daemonization has been moved from *-daemon.sh to the bin commands via the
--daemon option. Simply use --daemon start to start a daemon, --daemon stop to
stop a daemon, and --daemon status to set $? to the daemon's status. The
return code for status is LSB-compatible. For example, 'hdfs --daemon start
namenode'.
* It is now possible to override some of the shell code capabilities to provide
site specific functionality without replacing the shipped versions.
Replacement functions should go into the new hadoop-user-functions.sh file.
* A new option called --buildpaths will attempt to add developer build
directories to the classpath to allow for in source tree testing.
* Operations which trigger ssh connections can now use pdsh if installed.
${HADOOP_SSH_OPTS} still gets applied.
* Added distch and jnipath subcommands to the hadoop command.
* Shell scripts now support a --debug option which will report basic
information on the construction of various environment variables, java options,
classpath, etc. to help in configuration debugging.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP_SSH_PARALLEL} to prevent memory and network exhaustion.
By default, this is set to 10.
* ${HADOOP_CLIENT_OPTS} support has been added to a few more commands.
* Some subcommands were not listed in the usage.
* Various options on hadoop command lines were supported
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
Additionally, pid files are now created when daemons are run in interactive
mode. This will also prevent the accidental starting of two daemons with the
same configuration prior to launching java (i.e., fast fail without having to
wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly altered, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS_*, YARN_*, and MAPRED_* environment variables act as overrides to
their equivalent HADOOP_* environment variables when 'hdfs', 'yarn', 'mapred',
and related commands are executed. Previously, these were separated out which
meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files via *_OPT.
* Some formerly 'documented' entries in yarn-env.sh have been undocumented as a
simple form of deprecration in order to greatly simplify configuration and
reduce unnecessary duplication. They still work, but those variables will
likely be removed in a future release.
* Support for various undocumented YARN log4j.properties files has been removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented and unused yarn.id.str Java property has been removed.
* The unused yarn.policy.file Java property has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
* cygwin support has been removed.
NEW FEATURES:
* Daemonization has been moved from *-daemon.sh to the bin commands via the
--daemon option. Simply use --daemon start to start a daemon, --daemon stop to
stop a daemon, and --daemon status to set $? to the daemon's status. The
return code for status is LSB-compatible. For example, 'hdfs --daemon start
namenode'.
* It is now possible to override some of the shell code capabilities to provide
site specific functionality without replacing the shipped versions.
Replacement functions should go into the new hadoop-user-functions.sh file.
* A new option called --buildpaths will attempt to add developer build
directories to the classpath to allow for in source tree testing.
* Operations which trigger ssh connections can now use pdsh if installed.
${HADOOP_SSH_OPTS} still gets applied.
* Added distch and jnipath subcommands to the hadoop command.
* Shell scripts now support a --debug option which will report basic
information on the construction of various environment variables, java options,
classpath, etc. to help in configuration debugging.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
Additionally, pid files are now created when daemons are run in interactive
mode. This will also prevent the accidental starting of two daemons with the
same configuration prior to launching java (i.e., fast fail without having to
wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly altered, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS_*, YARN_*, and MAPRED_* environment variables act as overrides to
their equivalent HADOOP_* environment variables when 'hdfs', 'yarn', 'mapred',
and related commands are executed. Previously, these were separated out which
meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files via *_OPT.
* Support for various undocumented YARN log4j.properties files has been removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented and unused yarn.id.str Java property has been removed.
* The unused yarn.policy.file Java property has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
NEW FEATURES:
* Daemonization has been moved from *-daemon.sh to the bin commands via the
--daemon option. Simply use --daemon start to start a daemon, --daemon stop to
stop a daemon, and --daemon status to set $? to the daemon's status. The
return code for status is LSB-compatible. For example, 'hdfs --daemon start
namenode'.
* It is now possible to override some of the shell code capabilities to provide
site specific functionality without replacing the shipped versions.
Replacement functions should go into the new hadoop-user-functions.sh file.
* A new option called --buildpaths will attempt to add developer build
directories to the classpath to allow for in source tree testing.
* Operations which trigger ssh connections can now use pdsh if installed.
${HADOOP_SSH_OPTS} still gets applied.
* Added distch and jnipath subcommands to the hadoop command.
* Shell scripts now support a --debug option which will report basic
information on the construction of various environment variables, java options,
classpath, etc. to help in configuration debugging.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP_SSH_PARALLEL} to prevent memory and network exhaustion.
By default, this is set to 10.
* ${HADOOP_CLIENT_OPTS} support has been added to a few more commands.
* Some subcommands were not listed in the usage.
* Various options on hadoop command lines were supported
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
Additionally, pid files are now created when daemons are run in interactive
mode. This will also prevent the accidental starting of two daemons with the
same configuration prior to launching java (i.e., fast fail without having to
wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly altered, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS_*, YARN_*, and MAPRED_* environment variables act as overrides to
their equivalent HADOOP_* environment variables when 'hdfs', 'yarn', 'mapred',
and related commands are executed. Previously, these were separated out which
meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files via *_OPT.
* Some formerly 'documented' entries in yarn-env.sh have been undocumented as a
simple form of deprecration in order to greatly simplify configuration and
reduce unnecessary duplication. They still work, but those variables will
likely be removed in a future release.
* Support for various undocumented YARN log4j.properties files has been removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented and unused yarn.id.str Java property has been removed.
* The unused yarn.policy.file Java property has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
* cygwin support has been removed.
NEW FEATURES:
* Daemonization has been moved from *-daemon.sh to the bin commands via the
--daemon option. Simply use --daemon start to start a daemon, --daemon stop to
stop a daemon, and --daemon status to set $? to the daemon's status. The
return code for status is LSB-compatible. For example, 'hdfs --daemon start
namenode'.
* It is now possible to override some of the shell code capabilities to provide
site specific functionality without replacing the shipped versions.
Replacement functions should go into the new hadoop-user-functions.sh file.
* A new option called --buildpaths will attempt to add developer build
directories to the classpath to allow for in source tree testing.
* Operations which trigger ssh connections can now use pdsh if installed.
${HADOOP_SSH_OPTS} still gets applied.
* Added distch and jnipath subcommands to the hadoop command.
* Shell scripts now support a --debug option which will report basic
information on the construction of various environment variables, java options,
classpath, etc. to help in configuration debugging.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Issue Type: New Feature (was: Improvement) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
Additionally, pid files are now created when daemons are run in interactive
mode. This will also prevent the accidental starting of two daemons with the
same configuration prior to launching java (i.e., fast fail without having to
wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly alterated, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS_*, YARN_*, and MAPRED_* environment variables act as overrides to
their equivalent HADOOP_* environment variables when 'hdfs', 'yarn', 'mapred',
and related commands are executed. Previously, these were separated out which
meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files via *_OPT.
* Some formerly 'documented' entries in yarn-env.sh have been undocumented as a
simple form of deprecration in order to greatly simplify configuration and
reduce unnecessary duplication. They will still work, but those variables will
likely be removed in a future release.
* Support for various undocumentented YARN log4j.properties files has been
removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented yarn.id.str has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
* cygwin support has been removed.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP_SSH_PARALLEL} to prevent memory and network exhaustion.
By default, this is set to 10.
* ${HADOOP_CLIENT_OPTS} support has been added to a few more commands.
* Some subcommands were not listed in the usage.
* Various options on hadoop command lines were supported inconsistently. These
have been unified into hadoop-config.sh. --config is still required to be
first, however.
* ulimit logging for secure daemons no longer assumes /bin/bash but does assume
bash is on the command line path.
* Removed references to some Yahoo! specific paths.
* Removed unused slaves.sh from YARN build tree.
* Many exit states have been changed to reflect reality.
* Shell level errors now go to STDERR. Before, many of them went incorrectly
to STDOUT.
* CDPATH with a period (.) should no longer break the scripts.
* The scripts no longer try to chown directories.
IMPROVEMENTS:
* The *.out files are now appended instead of overwritten to allow for external
log rotation.
* The style and layout of the scripts is much more consistent across
subprojects.
* More of the shell code is now commented.
* Significant amounts of
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
Additionally, pid files are now created when daemons are run in interactive
mode. This will also prevent the accidental starting of two daemons with the
same configuration prior to launching java (i.e., fast fail without having to
wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly alterated, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS_*, YARN_*, and MAPRED_* environment variables act as overrides to
their equivalent HADOOP_* environment variables when 'hdfs', 'yarn', 'mapred',
and related commands are executed. Previously, these were separated out which
meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files via *_OPT.
* Some formerly 'documented' entries in yarn-env.sh have been undocumented as a
simple form of deprecration in order to greatly simplify configuration and
reduce unnecessary duplication. They will still work, but those variables will
likely be removed in a future release.
* Support for various undocumentented YARN log4j.properties files has been
removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented yarn.id.str has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP_SSH_PARALLEL} to prevent memory and network exhaustion.
By default, this is set to 10.
* ${HADOOP_CLIENT_OPTS} support has been added to a few more commands.
* Some subcommands were not listed in the usage.
* Various options on hadoop command lines were supported inconsistently. These
have been unified into hadoop-config.sh. --config is still required to be
first, however.
* ulimit logging for secure daemons no longer assumes /bin/bash but does assume
bash is on the command line path.
* Removed references to some Yahoo! specific paths.
* Removed unused slaves.sh from YARN build tree.
* Many exit states have been changed to reflect reality.
* Shell level errors now go to STDERR. Before, many of them went incorrectly
to STDOUT.
* CDPATH with a period (.) should no longer break the scripts.
* The scripts no longer try to chown directories.
IMPROVEMENTS:
* The *.out files are now appended instead of overwritten to allow for external
log rotation.
* The style and layout of the scripts is much more consistent across
subprojects.
* More of the shell code is now commented.
* Significant amounts of redundant code have been moved
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Resolution: Fixed Status: Resolved (was: Patch Available) Commit to trunk svn rev 1618847. Closing. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-15.patch I'll commit this after a jenkins run. -15 fixes the missing line in the copyright in hadoop-config.sh. That's sort of important... Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Open (was: Patch Available) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Patch Available (was: Open) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Open (was: Patch Available) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Patch Available (was: Open) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-16.patch -16: re-deprecate the previously not deprecated but documented hadoop nfs3 and hadoop portmap subcommands Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-15.patch, HADOOP-9902-16.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Fix Version/s: 3.0.0 Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Fix For: 3.0.0 Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-14.patch 9902-14 changes: * We now error and exit if hadoop-functions.sh isn't found in HADOOP_LIBEXEC_DIR. * All of the standalone exec JAVAs have been moved into a new hadoop_java_exec to be used for non-daemons. This now has an eval in front of it so that paths with spaces work again for MOST commands (most sbin/* commands are still broken, however). It also fixes [~rvs]'s 2nd blocker issue. * HADOOP_OS_TYPE documented and made a standard var. * populate_slaves_file moved to functions.sh and renamed to hadoop_populate_slaves_file. * A few of the functions in hadoop-functions.sh are now marked as not user replaceable. * A few echo's changed to hadoop_errors. * Removed the useless empty check around HADOOP_POLICYFILE. * hadoop_os_tricks now uses HADOOP_OS_TYPE rather than it's custom one and the bindv6only var was made local. Also a note about HADOOP_ALLOW_IPV6 being dev-only was added in case curious eyes trying to figure out how to get past the IPv6 protection see it. :D hadoop-functions.sh is stll showing up in libexec for me with this patch, so not sure how to proceed on that issue. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Patch Available (was: Open) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-14.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-13.patch HADOOP-9902-13-branch-2.patch Rebase for current and fixes a bug introduced by HADOOP-10927. Additionally, it was requested I provide a branch-2 patch for people to try out. This was a 'lazy' backport: not well tested and probably includes commands that don't exist in branch-2. Clearly not meant to be committed. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: (was: HADOOP-9902-13-branch-2.patch) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-13-branch-2.patch Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-12.patch rebase for git rev d469993fe9b270d01c4cf4542bb701297996fbac Minor stuff: * add usage for keys, credential * dirs with spaces should work again (this is very very broken in trunk btw) * merged in hadoop classpath changes from HADOOP-10903 * probably worthwhile to point out that this repairs the defaults that HADOOP-10759 broke ... will commit on Friday. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
Additionally, pid files are now created when daemons are run in interactive
mode. This will also prevent the accidental starting of two daemons with the
same configuration prior to launching java (i.e., fast fail without having to
wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly alterated, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS_*, YARN_*, and MAPRED_* environment variables act as overrides to
their equivalent HADOOP_* environment variables when 'hdfs', 'yarn', 'mapred',
and related commands are executed. Previously, these were separated out which
meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files via *_OPT.
* Some formerly 'documented' entries in yarn-env.sh have been undocumented as a
simple form of deprecration in order to greatly simplify configuration and
reduce unnecessary duplication. They will still work, but those variables will
likely be removed in a future release.
* Support for various undocumentented YARN log4j.properties files has been
removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented yarn.id.str has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP_SSH_PARALLEL} to prevent memory and network exhaustion.
By default, this is set to 10.
* ${HADOOP_CLIENT_OPTS} support has been added to a few more commands.
* Some subcommands were not listed in the usage.
* Various options on hadoop command lines were supported inconsistently. These
have been unified into hadoop-config.sh. --config is still required to be
first, however.
* ulimit logging for secure daemons no longer assumes /bin/bash but does assume
bash is on the command line path.
* Removed references to some Yahoo! specific paths.
* Removed unused slaves.sh from YARN build tree.
* Many exit states have been changed to reflect reality.
* Shell level errors now go to STDERR. Before, many of them went incorrectly
to STDOUT.
* CDPATH with a period (.) should no longer break the scripts.
* The scripts no longer try to chown directories.
* Hadoop deployments where the directories have spaces should work again.
IMPROVEMENTS:
* The *.out files are now appended instead of overwritten to allow for external
log rotation.
* The style and layout of the scripts is much more consistent across
subprojects.
* More of the shell code is
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
Additionally, pid files are now created when daemons are run in interactive
mode. This will also prevent the accidental starting of two daemons with the
same configuration prior to launching java (i.e., fast fail without having to
wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly alterated, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS_*, YARN_*, and MAPRED_* environment variables act as overrides to
their equivalent HADOOP_* environment variables when 'hdfs', 'yarn', 'mapred',
and related commands are executed. Previously, these were separated out which
meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files via *_OPT.
* Some formerly 'documented' entries in yarn-env.sh have been undocumented as a
simple form of deprecration in order to greatly simplify configuration and
reduce unnecessary duplication. They will still work, but those variables will
likely be removed in a future release.
* Support for various undocumentented YARN log4j.properties files has been
removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented yarn.id.str has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP_SSH_PARALLEL} to prevent memory and network exhaustion.
By default, this is set to 10.
* ${HADOOP_CLIENT_OPTS} support has been added to a few more commands.
* Some subcommands were not listed in the usage.
* Various options on hadoop command lines were supported inconsistently. These
have been unified into hadoop-config.sh. --config is still required to be
first, however.
* ulimit logging for secure daemons no longer assumes /bin/bash but does assume
bash is on the command line path.
* Removed references to some Yahoo! specific paths.
* Removed unused slaves.sh from YARN build tree.
* Many exit states have been changed to reflect reality.
* Shell level errors now go to STDERR. Before, many of them went incorrectly
to STDOUT.
* CDPATH with a period (.) should no longer break the scripts.
* The scripts no longer try to chown directories.
IMPROVEMENTS:
* The *.out files are now appended instead of overwritten to allow for external
log rotation.
* The style and layout of the scripts is much more consistent across
subprojects.
* More of the shell code is now commented.
* Significant amounts of redundant code have been moved
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-11.patch #11 - This is essentially the same patch, but incorporates two trivial changes that I missed/forgot to fix: HADOOP-6746 and HADOOP-6851. The latter has been filed in different forms for several years under a variety of different JIRAs. It has also been recommended by a PMC member that I give this patch a week and if no issues pop up, to just commit it to trunk. So let's start the clock on that. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-10.patch #10: Relatively minor bug fixes, enhanced error checking for log and pid dirs, more style cleanup, start/stop-dfs message consistency. Also, this has been run through (more or less) the hadoop smoke tests from Bigtop trunk. For some reason, some tests didn't work, such as the append test. I didn't bother troubleshooting those. The non-admin CLI tests that failed under bigtop I was able to run manually, so I suspect they failed for other, environmental reasons. Admin CLI tests passed. (For example, the MR tests failed because it looks for MR in /usr/lib/hadoop-mapreduce, despite having the MR home defined. But I can run, e.g., the pi example just fine using the exact same command it has in the test framework.) It'd still be great if someone could run this patch through the paces though. I'm running out of things to fix... ;) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-10.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
Additionally, pid files are now created when daemons are run in interactive
mode. This will also prevent the accidental starting of two daemons with the
same configuration prior to launching java (i.e., fast fail without having to
wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly alterated, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS_*, YARN_*, and MAPRED_* environment variables act as overrides to
their equivalent HADOOP_* environment variables when 'hdfs', 'yarn', 'mapred',
and related commands are executed. Previously, these were separated out which
meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files via *_OPT.
* Some formerly 'documented' entries in yarn-env.sh have been undocumented as a
simple form of deprecration in order to greatly simplify configuration and
reduce unnecessary duplication. They will still work, but those variables will
likely be removed in a future release.
* Support for various undocumentented YARN log4j.properties files has been
removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented yarn.id.str has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP_SSH_PARALLEL} to prevent memory and network exhaustion.
By default, this is set to 10.
* ${HADOOP_CLIENT_OPTS} support has been added to a few more commands.
* Some subcommands were not listed in the usage.
* Various options on hadoop command lines were supported inconsistently. These
have been unified into hadoop-config.sh. --config is still required to be
first, however.
* ulimit logging for secure daemons no longer assumes /bin/bash but does assume
bash is on the command line path.
* Removed references to some Yahoo! specific paths.
* Removed unused slaves.sh from YARN build tree.
* Many exit states have been changed to reflect reality.
* Shell level errors now go to STDERR. Before, many of them went incorrectly
to STDOUT.
IMPROVEMENTS:
* The style and layout of the scripts is much more consistent across
subprojects.
* More of the shell code is now commented.
* Significant amounts of redundant code have been moved into a new file called
hadoop-functions.sh.
* The various *-env.sh have been massively changed to include documentation and
examples on what can be set, ramifications of setting, etc. for all variables
that
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
Additionally, pid files are now created when daemons are run in interactive
mode. This will also prevent the accidental starting of two daemons with the
same configuration prior to launching java (i.e., fast fail without having to
wait for socket opening).
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly alterated, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All HDFS_*, YARN_*, and MAPRED_* environment variables act as overrides to
their equivalent HADOOP_* environment variables when 'hdfs', 'yarn', 'mapred',
and related commands are executed. Previously, these were separated out which
meant a significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files via *_OPT.
* Some formerly 'documented' entries in yarn-env.sh have been undocumented as a
simple form of deprecration in order to greatly simplify configuration and
reduce unnecessary duplication. They will still work, but those variables will
likely be removed in a future release.
* Support for various undocumentented YARN log4j.properties files has been
removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented yarn.id.str has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere, without requiring
symlinking and other such tricks.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the extended Hadoop ecosystem may still require this environment variable to be
configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP_SSH_PARALLEL} to prevent memory and network exhaustion.
By default, this is set to 10.
* ${HADOOP_CLIENT_OPTS} support has been added to a few more commands.
* Some subcommands were not listed in the usage.
* Various options on hadoop command lines were supported inconsistently. These
have been unified into hadoop-config.sh. --config is still required to be
first, however.
* ulimit logging for secure daemons no longer assumes /bin/bash but does assume
bash is on the command line path.
* Removed references to some Yahoo! specific paths.
* Removed unused slaves.sh from YARN build tree.
* Many exit states have been changed to reflect reality.
* Shell level errors now go to STDERR. Before, many of them went incorrectly
to STDOUT.
IMPROVEMENTS:
* The style and layout of the scripts is much most consistent across
subprojects.
* Much more of the shell code is now commented.
* Significant amounts of redundant code have been moved into a new file called
hadoop-functions.sh.
* The various *-env.sh have been massively changed to include documentation and
examples on what can be set, ramifications of setting, etc. for all variables
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid and out files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
* All Hadoop shell script subsystems now execute hadoop-env.sh, which allows
for all of the environment variables to be in one location. This was not the
case previously.
* The default content of *-env.sh has been significantly alterated, with the
majority of defaults moved into more protected areas inside the code.
Additionally, these files do not auto-append anymore; setting a variable on the
command line prior to calling a shell command must contain the entire content,
not just any extra settings. This brings Hadoop more in-line with the vast
majority of other software packages.
* All YARN_* and MAPRED_* environment variables act as overrides to their
equivalent HADOOP_* environment variables when 'yarn', 'mapred' and related
commands are executed. Previously, these were separated out which meant a
significant amount of duplication of common settings. * hdfs-config.sh and
hdfs-config.cmd were inadvertently duplicated into libexec and sbin. The sbin
versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files, in
particular via *_OPT.
* Some formerly documented entries in yarn-env.sh have been undocumented as a
simple form of deprecration in order to greatly simplify configuration and
reduce unnecessary duplication. They will still work, but those variables will
likely be removed in a future release.
* Support for various undocumentented YARN log4j.properties files has been
removed.
* Support for ${HADOOP_MASTER} and the related rsync code have been removed.
* The undocumented yarn.id.str has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere.
* ${HADOOP_CONF_DIR}/hadoop-layout.sh is now documented with a provided
hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path,
without ${HADOOP_PREFIX} being defined, and as the target of bash -x for
debugging. If ${HADOOP_PREFIX} is not set, it will be automatically determined
based upon the current location of the shell library. Note that other parts of
the ecosystem may require this environment variable to be configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP_SSH_PARALLEL} to prevent memory and network exhaustion.
By default, this is set to 10.
* ${HADOOP_CLIENT_OPTS} support has been added to a few more commands.
* Some subcommands were not listed in the usage.
* Various options on hadoop command lines were supported inconsistently. These
have been unified into hadoop-config.sh. --config still needs to come first,
however.
* ulimit logging for secure daemons no longer assumes /bin/bash but does assume
bash on the command line path.
* Removed references to some Yahoo! specific paths.
* Removed unused slaves.sh from YARN build tree.
* Many exit states have been changed to reflect reality.
* Shell level errors now go to STDERR. Before, many of them went incorrectly
to STDOUT.
*
IMPROVEMENTS:
* The style and layout of the scripts is now the same. This includes a vast
increase in code comments.
* Significant amounts of redundant code have been moved into a new file called
hadoop-functions.sh.
* The various *-env.sh have been massively changed to include documentation and
examples on what can be set, ramifications of setting, etc. for all variables
that are expected to be set by a user.
* There is now some trivial deduplication and sanitization of the classpath and
JVM options. This allows, amongst other things, for custom settings in *_OPTS
for Hadoop daemons to override defaults and other generic settings (i.e.,
${HADOOP_OPTS}). This is particularly relevant for Xmx settings, as one can
now set them in
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-9.patch Latest patch addresses the comments from the review. It also fixes some stylistic issues, cleans up some errors, and fixes a typo with lsSnapshotDir. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-8.patch a) fixed some bugs with HADOOP_SSH_OPTS b) proxyserver fell out of the usage information at some point c) resourcemanager -format-state-store got truncated d) minor whitespace fix Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-7.patch This version: * fixes the commit conflict with HADOOP-9921 * Documents a few missing env vars (that have been missing since those features were committed!) * Moves some defaults out of hadoop-env.sh into hdfs-config.sh so that hadoop-env.sh can run empty Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-6.patch vs -5, this patch fixes a kill output not sent to /dev/null, updates some of the hadoop-env.sh commentary, and fixes a code style issue in rotate_logs. If there is interest, I can make a patch for 2.4.1. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-4.patch Updated patch for git rev 8d5e8c860ed361ed792affcfe06f1a34b017e421. This includes many edge case bug fixes, a much more consistent coding style, the requested addition of the hadoop jnipath command, and a run through shellcheck. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Patch Available (was: Open) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Open (was: Patch Available) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Open (was: Patch Available) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-5.patch Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[
https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9902:
-
Release Note:
The Hadoop shell scripts have been rewritten to fix many long standing bugs and
include some new features. While an eye has been kept towards compatibility,
some changes may break existing installations.
INCOMPATIBLE CHANGES:
* The pid, out, etc files for secure daemons have been renamed to include the
appropriate ${HADOOP_IDENT_STR}. This should allow, with proper configurations
in place, for multiple versions of the same secure daemon to run on a host.
* All Hadoop shell script subsystems execute hadoop-env.sh, which allows for
all of the environment variables to be in one location. This was not the case
previously.
* The default content of *-env.sh has been significantly alterated, with the
majority of defaults moved into more protected areas.
* All YARN_* and MAPRED_* environment variables act as overrides to their
equivalent HADOOP_* environment variables when 'yarn', 'mapred' and related
commands are executed. Previously, these were separated out which meant a
significant amount of duplication of common settings.
* hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated into libexec
and sbin. The sbin versions have been removed.
* The log4j settings forcibly set by some *-daemon.sh commands have been
removed. These settings are now configurable in the *-env.sh files, in
particular via *_OPT.
* Support for various undocumentented YARN log4j.properties files has been
removed.
* Support for $HADOOP_MASTER and the related rsync code have been removed.
* yarn.id.str has been removed.
* We now require bash v3 (released July 27, 2004) or better in order to take
advantage of better regex handling and ${BASH_SOURCE}. POSIX sh will not work.
* Support for --script has been removed. We now use ${HADOOP_*_PATH} or
${HADOOP_PREFIX} to find the necessary binaries. (See other note regarding
${HADOOP_PREFIX} auto discovery.)
* Non-existent classpaths, ld.so library paths, JNI library paths, etc, will be
ignored and stripped from their respective environment settings.
BUG FIXES:
* ${HADOOP_CONF_DIR} is now properly honored everywhere.
* Documented hadoop-layout.sh with a provided hadoop-layout.sh.example file.
* Shell commands should now work properly when called as a relative path and
without HADOOP_PREFIX being defined. If ${HADOOP_PREFIX} is not set, it will be
automatically determined based upon the current location of the shell library.
Note that other parts of the ecosystem may require this environment variable to
be configured.
* Operations which trigger ssh will now limit the number of connections to run
in parallel to ${HADOOP_SSH_PARALLEL} to prevent memory and network exhaustion.
By default, this is set to 10.
* ${HADOOP_CLIENT_OPTS} support has been added to a few more commands.
* Various options on hadoop command lines were supported inconsistently. These
have been unified into hadoop-config.sh. --config still needs to come first,
however.
* ulimit logging for secure daemons no longer assumes /bin/bash but does assume
bash on the command line path.
* Removed references to some Yahoo! specific paths.
* Removed unused slaves.sh from YARN build tree.
IMPROVEMENTS:
* Significant amounts of redundant code have been moved into a new file called
hadoop-functions.sh.
* Improved information in the default *-env.sh on what can be set,
ramifications of setting, etc.
* There is an attempt to do some trivial deduplication and sanitization of the
classpath and JVM options. This allows, amongst other things, for custom
settings in *_OPTS for Hadoop daemons to override defaults and other generic
settings (i.e., $HADOOP_OPTS). This is particularly relevant for Xmx settings,
as one can now set them in _OPTS and ignore the heap specific options for
daemons which force the size in megabytes.
* Operations which trigger ssh connections can now use pdsh if installed.
$HADOOP_SSH_OPTS still gets applied.
* Subcommands have been alphabetized in both usage and in the code.
* All/most of the functionality provided by the sbin/* commands has been moved
to either their bin/ equivalents or made into functions. The rewritten
versions of these commands are now wrappers to maintain backward compatibility.
Of particular note is the new --daemon option present in some bin/ commands
which allow certain subcommands to be daemonized.
* It is now possible to override some of the shell code capabilities to provide
site specific functionality.
* A new option called --buildpaths will attempt to add developer build
directories to the classpath to allow for in source tree testing.
* If a usage function is defined, the following will trigger a help message if
it is given in the option path to the shell script: --? -? ? --help -help -h
help
*
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Patch Available (was: Open) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Release Note: The Hadoop shell scripts have been rewritten to fix many long standing bugs and include some new features. While an eye has been kept towards compatibility, some changes may break existing installations. INCOMPATIBLE CHANGES: * The pid files for secure daemons have been renamed to include the appropriate $HADOOP_IDENT_STR. This should allow, with proper configurations in place, for multiple versions of the same secure daemon to run on a host. * All YARN_* and MAPRED_* environment variables act as overrides to their equivalent HADOOP_* environment variables when 'yarn', 'mapred' and related commands are executed. Previously, these were separated which meant duplication of common settings. * All Hadoop shell script subsystems execute hadoop-env.sh, which allows for all of the environment variables to be in one location. This was not the case previously. * hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated placed into libexec and sbin during install. The sbin version has been removed. * The log4j settings forcibly set by some *-daemon.sh commands have been removed. This is now configurable in the *-env.sh files. Users who do not have these set will see logs going in odd places. * Support for various undocumentented YARN log4j.properties files has been removed. * Support for $HADOOP_MASTER and the related rsync code have been removed. * yarn.id.str has been removed. * We now require bash v3 (released July 27, 2004) or better in order to take advantage of better regex handling. * Support for --script has been removed from the sbin commands. We now use $HADOOP_*_PATH or $HADOOP_PREFIX to find the necessary binaries. BUG FIXES: * HADOOP_CONF_DIR is now properly honored everywhere. * Documented hadoop-layout.sh. * Added better comments to *-env.sh. * Shell commands should now work properly when called as a relative path and without HADOOP_PREFIX. If HADOOP_PREFIX is not set, it will be automatically determined based upon the current location of the shell command. Note that other parts of the ecosystem may require this environment variable to be configured. * Operations which trigger ssh will now limit how many connections run in parallel to 10 to prevent memory and network exhaustion. * HADOOP_CLIENT_OPTS support has been added to a few more commands. * Various options on hadoop command lines were supported inconsistently. These have been unified into hadoop-config.sh. --config still needs to come first, however. * ulimit logging for secure daemons no longer assumes /bin/bash but does assume bash on the command line path. * Removed references to some Yahoo! specific paths. * Removed unused slaves.sh from YARN build tree. IMPROVEMENTS: * Significant amounts of redundant code have been moved into a new file called hadoop-functions.sh. * Improved information in *-env.sh on what can be set, ramifications of setting, etc. * There is an attempt to do some trivial deduplication of the classpath and JVM options. This allows, amongst other things, for custom settings in *_OPTS for Hadoop daemons to override defaults and other generic settings (i.e., $HADOOP_OPTS). This is particularly relevant for Xmx settings, as one can now set them in _OPTS and ignore the heap specific options for daemons which force the size in megabytes. * Operations which trigger ssh connections can now use pdsh if installed. $HADOOP_SSH_OPTS still gets applied. * Subcommands have been alphabetized in both usage and in the code. * All/most of the functionality provided by the sbin/* commands has been moved to either their bin/ equivalents or made into functions. The rewritten versions of these commands are now wrappers to maintain backward compatibility. Of particular note is the new --daemon option present in some bin/ commands which allow certain subcommands to be daemonized. * It is now possible to override some of the shell code capabilities to provide site specific functionality. * A new option called --buildpaths will attempt to add developer build directories to the classpath to allow for in source tree testing. * If a usage function is defined, the following will trigger a help message if it is given in the option path to the shell script: --? -? ? --help -help -h help * Several generic environment variables have been added to provide a common configuration for pids, logs, and their security equivalents. The older versions still act as overrides to these generic versions. * Groundwork has been laid to allow for custom secure daemon setup using something other than jsvc. * Added distch subcommand to hadoop command. was: The Hadoop shell scripts have been rewritten to fix many long standing bugs and include some new features. While an eye
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Open (was: Patch Available) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Target Version/s: 3.0.0 Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-3.patch Great feedback! Here's a new patch that incorporates that change as well as fixing a ton of bugs (esp around secure daemons), has some code cleanup, env var namespace cleanup, and one or two minor new features. (See the release notes). I think I'm at the point where I need other people to start banging on this patch to test it out. The sooner we get it into trunk, the better. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902-3.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Release Note: The Hadoop shell scripts have been rewritten to fix many long standing bugs and include some new features. While an eye has been kept towards compatibility, some changes may break existing installations. INCOMPATIBLE CHANGES: * The pid files for secure daemons have been renamed to include the appropriate $HADOOP_IDENT_STR. This should allow, with proper configurations in place, for multiple versions of the same secure daemon to run on a host. * All YARN_* and MAPRED_* environment variables act as overrides to their equivalent HADOOP_* environment variables when 'yarn', 'mapred' and related commands are executed. Previously, these were separated which meant duplication of common settings. * All Hadoop shell script subsystems execute hadoop-env.sh, which allows for all of the environment variables to be in one location. This was not the case previously. * hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated in two different locations. The sbin version has been removed. * The log4j settings forcibly set by some *-daemon.sh commands has been removed. This is now configurable in the *-env.sh files. Users who do not have these set will see logs going in odd places. * Support for various undocumentented YARN log4j.properties files has been removed. * Support for $HADOOP_MASTER and the rsync code has been removed. * yarn.id.str has been removed. * We now require bash v3 or better. BUG FIXES: * HADOOP_CONF_DIR is now properly honored. * Documented hadoop-layout.sh * Shell commands should now work properly when called as a relative path. * Operations which trigger ssh will now limit how many connections run in parallel to 10 to prevent memory and network exhaustion. * HADOOP_CLIENT_OPTS support has been added to a few more commands. * Various options on hadoop comamnd lines were supported inconsistently. These have been unified into hadoop-config.sh. --config still needs to come first, however. * ulimit logging for secure daemons no longer assumes /bin/bash but does assume bash on the command line path. * Removed references to some Yahoo! specific paths. IMPROVEMENTS: * Significant amounts of redundant code has been moved into a new file called hadoop-functions.sh. * Improved information in *-env.sh on what can be set, ramifications of setting, etc. * There is an attempt to do some trivial deduplication of the classpath and JVM options. This allows, amongst other things, for custom settings in *_OPTS for Hadoop daemons to override defaults and other generic settings (i.e., $HADOOP_OPTS). This is particularly relevant for Xmx settings, as one can now set them in _OPTS and ignore the heap specific options for daemons which force the size in megabytes. * Operations which trigger ssh connections can now use pdsh if installed. $HADOOP_SSH_OPTS still gets applied. * Subcommands have been alphabetized in both usage and in the code. * All/most of the functionality provided by the sbin/* commands has been moved to either their bin/ equivalents or made into functions. The rewritten versions of these commands are now wrappers to maintain backward compatibility. Of particular note is the new --daemon option present in some bin/ commands which allow certain subcommands to be daemonized. * It is now possible to override some of the shell code capabilities to provide site specific functionality. * A new option called --buildpaths will attempt to add developer build directories to the classpath to allow for in source tree testing. * If a usage function is defined, -h, -help, and --help will all trigger a help message. * Several generic environment variables have been added to provide a common configuration for pids, logs, and their security equivalents. The older versions still act as overrides to these generic versions. * Groundwork has been laid to allow for custom secure daemon setup using something other than jsvc. Hadoop Flags: Incompatible change Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Release Note: The Hadoop shell scripts have been rewritten to fix many long standing bugs and include some new features. While an eye has been kept towards compatibility, some changes may break existing installations. INCOMPATIBLE CHANGES: * The pid files for secure daemons have been renamed to include the appropriate $HADOOP_IDENT_STR. This should allow, with proper configurations in place, for multiple versions of the same secure daemon to run on a host. * All YARN_* and MAPRED_* environment variables act as overrides to their equivalent HADOOP_* environment variables when 'yarn', 'mapred' and related commands are executed. Previously, these were separated which meant duplication of common settings. * All Hadoop shell script subsystems execute hadoop-env.sh, which allows for all of the environment variables to be in one location. This was not the case previously. * hdfs-config.sh and hdfs-config.cmd were inadvertently duplicated in two different locations. The sbin version has been removed. * The log4j settings forcibly set by some *-daemon.sh commands has been removed. This is now configurable in the *-env.sh files. Users who do not have these set will see logs going in odd places. * Support for various undocumentented YARN log4j.properties files has been removed. * Support for $HADOOP_MASTER and the rsync code have been removed. * yarn.id.str has been removed. * We now require bash v3 or better. BUG FIXES: * HADOOP_CONF_DIR is now properly honored. * Documented hadoop-layout.sh. * Shell commands should now work properly when called as a relative path. * Operations which trigger ssh will now limit how many connections run in parallel to 10 to prevent memory and network exhaustion. * HADOOP_CLIENT_OPTS support has been added to a few more commands. * Various options on hadoop command lines were supported inconsistently. These have been unified into hadoop-config.sh. --config still needs to come first, however. * ulimit logging for secure daemons no longer assumes /bin/bash but does assume bash on the command line path. * Removed references to some Yahoo! specific paths. IMPROVEMENTS: * Significant amounts of redundant code have been moved into a new file called hadoop-functions.sh. * Improved information in *-env.sh on what can be set, ramifications of setting, etc. * There is an attempt to do some trivial deduplication of the classpath and JVM options. This allows, amongst other things, for custom settings in *_OPTS for Hadoop daemons to override defaults and other generic settings (i.e., $HADOOP_OPTS). This is particularly relevant for Xmx settings, as one can now set them in _OPTS and ignore the heap specific options for daemons which force the size in megabytes. * Operations which trigger ssh connections can now use pdsh if installed. $HADOOP_SSH_OPTS still gets applied. * Subcommands have been alphabetized in both usage and in the code. * All/most of the functionality provided by the sbin/* commands has been moved to either their bin/ equivalents or made into functions. The rewritten versions of these commands are now wrappers to maintain backward compatibility. Of particular note is the new --daemon option present in some bin/ commands which allow certain subcommands to be daemonized. * It is now possible to override some of the shell code capabilities to provide site specific functionality. * A new option called --buildpaths will attempt to add developer build directories to the classpath to allow for in source tree testing. * If a usage function is defined, -h, -help, and --help will all trigger a help message. * Several generic environment variables have been added to provide a common configuration for pids, logs, and their security equivalents. The older versions still act as overrides to these generic versions. * Groundwork has been laid to allow for custom secure daemon setup using something other than jsvc. was: The Hadoop shell scripts have been rewritten to fix many long standing bugs and include some new features. While an eye has been kept towards compatibility, some changes may break existing installations. INCOMPATIBLE CHANGES: * The pid files for secure daemons have been renamed to include the appropriate $HADOOP_IDENT_STR. This should allow, with proper configurations in place, for multiple versions of the same secure daemon to run on a host. * All YARN_* and MAPRED_* environment variables act as overrides to their equivalent HADOOP_* environment variables when 'yarn', 'mapred' and related commands are executed. Previously, these were separated which meant duplication of common settings. * All Hadoop shell script subsystems execute hadoop-env.sh, which allows for all of the environment variables to be
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902-2.patch Should apply to git commit faf2d78012fd6fcf5fc433ab85b2dbc9d672c125 Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-9902-2.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Labels: releasenotes (was: ) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Labels: releasenotes (was: in releasenotes) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Labels: in releasenotes (was: releasenotes) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Patch Available (was: Open) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: releasenotes Attachments: HADOOP-9902-2.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902.patch Here's the latest version of this patch against trunk (svn revision 1598750). There are still a few things I want to fix and I'm sure there are bugs floating around here and there. I'd appreciate any feedback with the patch thus far! Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: (was: HADOOP-9902.patch) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902.patch Let's try this again: git rev ca2d0153bf3ec2f7f228bb1e68c0cadf4fb2d6c5 svn rev 1598764 Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Affects Version/s: (was: 2.1.1-beta) Status: Patch Available (was: Open) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: HADOOP-9902.patch Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Status: Open (was: Patch Available) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: (was: HADOOP-9902.patch) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Affects Version/s: 3.0.0 Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0, 2.1.1-beta Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: (was: scripts.tgz) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.1.1-beta Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: hadoop-9902-1.patch Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.1.1-beta Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: hadoop-9902-1.patch, more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: scripts.tgz Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.1.1-beta Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: more-info.txt, scripts.tgz, scripts.tgz Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: (was: scripts.tgz) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.1.1-beta Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: more-info.txt, scripts.tgz Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Description: Umbrella JIRA for shell script rewrite. See more-info.txt for more details. (was: Umbrella JIRA for shell script rewrite. See first comment for more details.) Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.1.1-beta Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: more-info.txt Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.1.1-beta Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: more-info.txt Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9902) Shell script rewrite
[ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9902: - Attachment: scripts.tgz Just to give an idea of what I'm thinking, here is a sample. Note this is a) not even close to final, b) likely has bugs, c) is very incomplete, and d) hasn't been fully optimized at all. This is for 2.1.0. Sorry for not being in patch format, but I'm not at that stage yet. Shell script rewrite Key: HADOOP-9902 URL: https://issues.apache.org/jira/browse/HADOOP-9902 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.1.1-beta Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: more-info.txt, scripts.tgz Umbrella JIRA for shell script rewrite. See more-info.txt for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
