Repository: impala
Updated Branches:
  refs/heads/master 4c1538ab1 -> 830e3346f


Remove Yarn from minicluster by default.

Turns out that we start Yarn as part of the minicluster, but we never use it.
(HiveServer2 is configured to run MR jobs "locally" in process.) Likely, this
Yarn integration is a vestige of Yarn/Llama integration.  We can save memory by
not starting it by default.

There are some less-common tooks like tests/comparison/cluster.py which use
Yarn (and Hadoop Streaming). In deference to those tools, I've left a mechanism
to start Yarn rather than excising it altogether. After running
buildall the regular way, add Yarn to the cluster by running:
  testdata/cluster/admin -y start_cluster

I tested by running core tests. I did not test the kerberized minicluster.

Change-Id: I5504cc40b89e3c6d53fac0b7aa4b395fa63e8d79


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/942781d8
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/942781d8
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/942781d8

Branch: refs/heads/master
Commit: 942781d80f243d36c557edb67ad828e8be5ff0d5
Parents: 4c1538a
Author: Philip Zeyliger <phi...@cloudera.com>
Authored: Mon Apr 9 15:10:43 2018 -0700
Committer: Philip Zeyliger <phi...@cloudera.com>
Committed: Tue Apr 10 09:17:28 2018 -0700

----------------------------------------------------------------------
 testdata/cluster/admin | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/942781d8/testdata/cluster/admin
----------------------------------------------------------------------
diff --git a/testdata/cluster/admin b/testdata/cluster/admin
index cb33e21..74b5a9c 100755
--- a/testdata/cluster/admin
+++ b/testdata/cluster/admin
@@ -18,7 +18,7 @@
 # specific language governing permissions and limitations
 # under the License.
 
-# This will create/control/destroy a local hdfs+yarn cluster.
+# This will create/control/destroy a local hdfs/yarn/kms/kudu cluster.
 #
 # All roles run on 127.0.0.1, just like the standard mini cluster included 
with hadoop.
 # The difference is with this cluster, each role runs in its own process and 
has its own
@@ -31,12 +31,14 @@ set -euo pipefail
 trap 'echo Error in $0 at line $LINENO: $(awk "NR == $LINENO" $0)' ERR
 
 : ${IMPALA_KERBERIZE=}
+: ${INCLUDE_YARN=}
 
-while getopts vk OPT; do
+while getopts vky OPT; do
   case $OPT in
     v) set -x;;
     k) export IMPALA_KERBERIZE=1;;
-    ?) echo "Usage: $0 [-v (verbose) -k (kerberize)] ACTION (see source...)"; 
exit 1;;
+    y) export INCLUDE_YARN=1;;
+    ?) echo "Usage: $0 [-v (verbose) -k (kerberize) -y (yarn)] ACTION (see 
source...)"; exit 1;;
   esac
 done
 shift $(($OPTIND-1))
@@ -54,7 +56,10 @@ export KILL_CLUSTER_MARKER=IBelongToTheMiniCluster
 
 if [[ "$TARGET_FILESYSTEM" == "hdfs" ]]; then
   # The check above indicates that the regular mini-cluster is in use.
-  SUPPORTED_SERVICES=(hdfs yarn kms)
+  SUPPORTED_SERVICES=(hdfs kms)
+  if [ -n "${INCLUDE_YARN}" ]; then
+    SUPPORTED_SERVICES+=(yarn)
+  fi
 else
   # Either a remote distributed file system or a local non-distributed file 
system is
   # in use. Currently the only service that is expected to work is Kudu, 
though in theory

Reply via email to