On 21 November 2012 19:15, Matt Foley ma...@apache.org wrote:
This discussion started in
Those of us involved in the branch-1-win port of Hadoop to Windows without
use of Cygwin, have faced the issue of frequent use of shell scripts
throughout the system, both in build time (eg, the utility
saveVersion.sh),
and run time (config files like hadoop-env.sh and the start/stop scripts
in bin/* ). Similar usages exist throughout the Hadoop stack, in all
projects.
The vast majority of these shell scripts do not do anything platform
specific; they can be expressed in a posix-conforming way. Therefore, it
seems to us that it makes sense to start using a cross-platform scripting
language, such as python, in place of shell for these purposes. For those
rare occasions where platform-specific functionality really is needed,
python also supports quite a lot of platform-specific functionality on both
Linux and Windows; but where that is inadequate, one could still
conditionally invoke a platform-specific module written in shell (for
Linux/*nix) or powershell or bat (for Windows).
The primary motive for moving to a cross-platform scripting language is
maintainability. The alternative would be to maintain two complete suites
of scripts, one for Linux and one for Windows (and perhaps others in the
future). We want to avoid the need to update dual modules in two different
languages when functionality changes, especially given that many Linux
developers are not familiar with powershell or bat, and many Windows
developers are not familiar with shell or bash.
I'd argue that a lot of Hadoop java developers aren't that familiar with
bash. It's only in the last six months that I've come to hate it properly.
In the ant project, it was the launcher scripts that had the worst
bugrep:line ratio, as
-variations in .sh behaviour, especially under cygwin, but also things
that weren't bash (AIX, ...)
-requirements of the entire unix command set for real work
-variants in the parameters/behaviour of those commands between Linux and
other widely used Unix systems (e.g. OSX)
-lack of inclusion of the .sh scripts in the junit test suite
-lack of understanding of bash.
In the ant project we added a Python launcher in, what, 2001, based on the
Perl launcher supplied by one steve_l@users.sourceforge
For run-time, there is likely to be a lot more discussion. Lots of folks,
including me, aren't real happy with use of active scripts for
configuration, and various others, including I believe some of the Bigtop
folks, have issues with the way the start/stop scripts work. Nevertheless,
all those scripts exist today and are widely used. And they present an
impediment to porting to Windows-without-cygwin.
They're a maintenance and support cost on Unix. Too many scripts, even more
in Yarn, weakly-nondeterministic logic for loading env variables,
especially between init.d and bin/hadoop; not much diagnostics. And as with
Ant, a relatively under-comprehended language with no unit test coverage.
I'd replace the bash logic with python for Unix dev and maintenance alone.
You could put your logic into a shared python module in usr/lib/hadoop/bin
, have PyUnit test the inner functions as part of the build and test
process ( jenkins).
Nothing about run-time use of scripts has changed significantly over the
past three years, and I don't think we should hold up the Windows port
while we have a huge discussion about issues that veer dangerously into
religious/aesthetic domains. It would be fun to have that discussion, but I
don't want this decision to be dependent on it!
With Yarn its got more complex. More env variables to set, more support
calls when they aren't.
So I propose that we go ahead and also approve python as a run-time
dependency, and allow the inclusion of python scripts in place of current
shell-based functionality. The unpleasant alternative is to spawn a bunch
of powershell scripts in parallel to the current shell scripts, with a very
negative impact on maintainability. The Windows port must, after all, be
allowed to proceed.
+1 to any vote to allow .py at run time as a new feature
=0 to ripping out and replacing the existing .sh scripts with python code,
as even though I don't like the scripts, replacing them could be traumatic
downstream.
+1 to a gradual migration to .py for new code, starting with the yarn
scripts.
