IMPALA-4365: Enabling end-to-end tests on a remote cluster
This patch lays the groundwork for loading data and running end-to-end
tests on a remote CDH cluster. The requirements for the cluster to run
the tests are:
- Managed by Cloudera Manager (CM)
- GPL Extras need to be installed
- KMS and KeyTrustee installed and available as a service
- SERDEPROPERTIES in the Hive DB modified to accept wide tables
- Hive warehouse dir points to /test-warehouse
The actual data loading is done via a new script, remote_data_load.py,
which takes the CM host as an argument. It can be run from a client
machine that is not a node of the cluster, but it needs to have the
Impala repo checked out and Impala built. This insures that all of the
necessary data load scripts are available, as well as setting up the
environment properly (client binaries like beeline and the hbase shell
are available, python libraries like cm_api are installed, necessary
environment variables are defined, etc.)
It should be noted that running remote_data_load.py will overwrite
any local XML config files with the configurations downloaded from
the remote cluster.
Usage: remote_data_load.py [options] <cm_host address>
Options:
-h, --help show this help message and exit
--snapshot-file=SNAPSHOT_FILE
Path to the test-warehouse archive
--cm-user=CM_USER Cloudera Manager admin user
--cm-pass=CM_PASS Cloudera Manager admin user password
--gateway=GATEWAY Gateway host to upload the data from. If not
set, uses the CM host as gateway.
--ssh-user=SSH_USER System user on the remote machine with
passwordless SSH configured.
--no-load Do not try to load the snapshot
--exploration-strategy=EXPLORATION_STRATEGY
--test Run end-to-end tests against cluster
Testing:
This patch is being submitted with the understanding that there are
still clean up issues that need to be addressed in the remote data
load script, for which JIRA's have been filed.
However, since many of the existing build scripts also had to be
modified, it is more important to make sure that no regressions were
inadvertently introduced into the existing data load process. Loading
data to a local mini-cluster was checked repeatedly while this patch
was being developed, as well as running it against the Jenkins job
that provides the test-warehouse snapshot used by the many other
Impala CI builds that run daily.
Change-Id: I1f443a1728a1d28168090c6f54e82dec2cb073e9
Reviewed-on: http://gerrit.cloudera.org:8080/4769
Reviewed-by: Taras Bobrovytsky <[email protected]>
Tested-by: Internal Jenkins
Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/ce4c5f67
Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/ce4c5f67
Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/ce4c5f67
Branch: refs/heads/hadoop-next
Commit: ce4c5f67433ebe050261710920972621d625c81c
Parents: ef689ed
Author: Martin Grund <[email protected]>
Authored: Wed Aug 19 22:18:47 2015 -0700
Committer: Internal Jenkins <[email protected]>
Committed: Tue Nov 8 10:16:55 2016 +0000
----------------------------------------------------------------------
bin/load-data.py | 12 +
bin/remote_data_load.py | 560 +++++++++++++++++++
testdata/bin/compute-table-stats.sh | 8 +-
testdata/bin/create-load-data.sh | 176 ++++--
testdata/bin/create-table-many-blocks.sh | 14 +-
testdata/bin/generate-schema-statements.py | 3 +-
testdata/bin/load-test-warehouse-snapshot.sh | 37 +-
testdata/bin/load_nested.py | 13 +-
testdata/bin/run-step.sh | 1 +
testdata/bin/setup-hdfs-env.sh | 32 +-
.../datasets/functional/schema_constraints.csv | 5 +
11 files changed, 797 insertions(+), 64 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ce4c5f67/bin/load-data.py
----------------------------------------------------------------------
diff --git a/bin/load-data.py b/bin/load-data.py
index df41786..1359e01 100755
--- a/bin/load-data.py
+++ b/bin/load-data.py
@@ -22,6 +22,7 @@
# Most ddl commands are executed by Impala.
import collections
import getpass
+import logging
import os
import re
import sqlparse
@@ -30,12 +31,17 @@ import sys
import tempfile
import time
import traceback
+
from itertools import product
from optparse import OptionParser
from Queue import Queue
from tests.beeswax.impala_beeswax import *
from threading import Thread
+logging.basicConfig()
+LOG = logging.getLogger('load-data.py')
+LOG.setLevel(logging.DEBUG)
+
parser = OptionParser()
parser.add_option("-e", "--exploration_strategy", dest="exploration_strategy",
default="core",
@@ -191,6 +197,7 @@ def generate_schema_statements(workload):
generate_cmd += " --hive_warehouse_dir=%s" % options.hive_warehouse_dir
if options.hdfs_namenode is not None:
generate_cmd += " --hdfs_namenode=%s" % options.hdfs_namenode
+ generate_cmd += " --backend=%s" % options.impalad
print 'Executing Generate Schema Command: ' + generate_cmd
schema_cmd = os.path.join(TESTDATA_BIN_DIR, generate_cmd)
error_msg = 'Error generating schema statements for workload: ' + workload
@@ -265,6 +272,11 @@ def invalidate_impala_metadata():
impala_client.close_connection()
if __name__ == "__main__":
+ # Having the actual command line at the top of each data-load-* log can help
+ # when debugging dataload issues.
+ #
+ LOG.debug(' '.join(sys.argv))
+
all_workloads = available_workloads(WORKLOAD_DIR)
workloads = []
if options.workloads is None:
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ce4c5f67/bin/remote_data_load.py
----------------------------------------------------------------------
diff --git a/bin/remote_data_load.py b/bin/remote_data_load.py
new file mode 100755
index 0000000..85a9f95
--- /dev/null
+++ b/bin/remote_data_load.py
@@ -0,0 +1,560 @@
+#!/usr/bin/env python
+# Copyright 2015 Cloudera Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# This is a setup script that will downloaded a test warehouse snapshot and
+# deploy it on a remote, CM-managed cluster. Once the data is loaded, it is
+# possible to run a subset of the Impala core / exhaustive tests on the
+# remote cluster.
+#
+# * This script should be executed from a machine that has the Impala
+# development environment set up.
+#
+# * The cluster needs to be configured appropriately:
+# - The following services need to be installed:
+# HDFS, YARN, HIVE, IMPALA, MAPREDUCE, KUDU, HBASE, ZOOKEEPER
+# - GPL Extras parcel needs to be installed
+# - Metastore DB SERDE properties PARAM_VALUE needs to be altered to
+# allow for wide tables (See HIVE-1364.)
+# - The hive-warehouse path needs to be /test-warehouse
+#
+# Usage: remote_data_load.py [options] cm_host
+#
+# Options:
+# -h, --help show this help message and exit
+# --cm_user=CM_USER Cloudera Manager admin user
+# --cm_pass=CM_PASS Cloudera Manager admin user password
+# --no-load Do not try to load the snapshot
+# --test Run end-to-end tests against cluster.
+# --gateway=GATEWAY Gateway host to upload the data from. If not set,
uses
+# the CM host as gateway.
+# --ssh_user=SSH_USER System user on the remote machine with passwordless
SSH
+# configured.
+#
+import fnmatch
+import glob
+import logging
+import os
+import sh
+import shutil
+import sys
+import time
+
+from cm_api.api_client import ApiResource
+from functools import wraps
+from optparse import OptionParser
+from sh import ssh
+from tempfile import mkdtemp
+from urllib import quote as urlquote
+
+
+REQUIRED_SERVICES = ['HBASE',
+ 'HDFS',
+ 'HIVE',
+ 'IMPALA',
+ 'KUDU',
+ 'MAPREDUCE',
+ 'YARN',
+ 'ZOOKEEPER']
+
+# TODO: It's not currently possible to get the version from the cluster.
+# It would be nice to generate this dynamically.
+# (v14 happens to be the version that ships with CDH 5.9.x)
+CM_API_VERSION = 'v14'
+
+# Impala's data loading and test framework assumes this Hive Warehouse
Directory.
+# Making this configurable would be an invasive change, and therefore, we
prefer to
+# re-configure the Hive service via the CM API before loading data and running
tests.
+HIVE_WAREHOUSE_DIR = "/test-warehouse"
+
+logger = logging.getLogger("remote_data_load")
+logger.setLevel(logging.DEBUG)
+
+# Goes to the file
+fh = logging.FileHandler("remote_data_load.log")
+fh.setLevel(logging.DEBUG)
+
+# Goes to stdout
+ch = logging.StreamHandler()
+ch.setLevel(logging.INFO)
+
+# create formatter and add it to the handlers
+formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s -
%(message)s')
+fh.setFormatter(formatter)
+ch.setFormatter(formatter)
+
+# add the handlers to the logger
+logger.addHandler(fh)
+logger.addHandler(ch)
+
+
+def timing(func):
+ """
+ A decorator for timing how much time a function takes.
+
+ We can modify this later to do something more intelligent than just
logging.
+ """
+ @wraps(func)
+ def wrap(*args, **kwargs):
+ t1 = time.time()
+ result = func(*args, **kwargs)
+ t2 = time.time()
+
+ output = 'TIME: {name}() took: {t:.4f} seconds'
+ logger.info(output.format(name=func.__name__, t=(t2-t1)))
+ return result
+ return wrap
+
+
+def tee(line):
+ """Output wrapper function used by sh to send the stdout or stderr to the
+ module's logger."""
+ logger.debug(line.strip())
+
+
+class RemoteDataLoad(object):
+ """This is an implementation of the process to load a test-warehouse
snapshot on
+ a remote CM managed cluster. This script assumes that the warehouse
snapshot was
+ already downloaded and was either passed in as a parameter, or can be
found by
+ either inspecting the SNAPSHOT_DIR environment variable, or based on the
WORKSPACE
+ environment variable on a Jenkins build slave.
+
+ The reason for the additional setup code is that in the local development
+ environment it is assumed that $USER is HDFS superuser, which is not the
case for
+ remote deloyments.
+ """
+
+ def __init__(self, cm_host, options):
+ logger.info("Starting remote data load...")
+ self.options = options
+ self.cm_host = cm_host
+
+ # Gateway host can be used if the CM host is not configured as a
Hadoop gateway
+ self.gateway = options.gateway if options.gateway else cm_host
+ self.impala_home = os.environ["IMPALA_HOME"]
+ self.api = ApiResource(self.cm_host, username=options.cm_user,
+ password=options.cm_pass)
+
+ # The API returns a list of clusters managed by the CM host. We're
assuming
+ # that this CM host was set up for the purpose of Impala testing on one
+ # cluster, so the list should only have one value.
+ self.cluster = self.api.get_all_clusters()[0]
+ self.services = self.get_services()
+
+ self.config = self.get_service_client_configurations()
+ logger.info("Retrieved service configuration")
+ logger.info(str(self.config))
+ self.prepare()
+ logger.info("IMPALA_HOME: {0}".format(self.impala_home))
+
+ def get_hostname_for_ref(self, host_ref):
+ """Translate the HostRef instance into the hostname."""
+ return self.api.get_host(host_ref.hostId).hostname
+
+ @staticmethod
+ def get_or_default(config):
+ return config.value if config.value else config.default
+
+ def get_services(self):
+ """Confirm that all services are running, and return service dict."""
+ services = dict((s.type, s) for s in self.cluster.get_all_services())
+
+ if set(REQUIRED_SERVICES) != set(services.keys()):
+ missing_services = set(REQUIRED_SERVICES) - set(services.keys())
+ logger.error("Services not installed:
{0}".format(list(missing_services)))
+ raise RuntimeError("Cluster not ready.")
+
+ if not all(services[s].serviceState == 'STARTED' for s in services):
+ stopped = [s for s in services if services[s].serviceState !=
"STARTED"]
+ logger.error("Not all services started: {0}".format(stopped))
+ raise RuntimeError("Cluster not ready.")
+
+ return services
+
+ @timing
+ def download_client_config(self, cluster, service):
+ """Download the client configuration zip for a particular cluster and
service.
+
+ Since cm_api does not provide a way to download the archive we build
the URL
+ manually and download the file. Once it downloaded the file the
archive is
+ extracted and its content is copied to the Hadoop configuration
directories
+ defined by Impala.
+ """
+ logger.info("Downloading client configuration for
{0}".format(service.name))
+ url =
"http://{0}:7180/api/{1}/clusters/{2}/services/{3}/clientConfig".format(
+ self.cm_host, CM_API_VERSION, urlquote(cluster.name),
urlquote(service.name))
+ path = mkdtemp()
+ sh.curl(url, o=os.path.join(path, "clientConfig.zip"), _out=tee,
_err=tee)
+ current = os.getcwd()
+ os.chdir(path)
+ sh.unzip("clientConfig.zip")
+ for root, _, file_names in os.walk("."):
+ for filename in fnmatch.filter(file_names, "*.xml"):
+ src = os.path.join(root, filename)
+ dst = os.path.join(self.impala_home, "fe", "src", "test",
"resources")
+ logger.debug("Copying {0} to {1}".format(src, dst))
+ shutil.copy(src, dst)
+ os.chdir(current)
+
+ # TODO: this may be available in tests/comparison/cluster.py
+ def set_hive_warehouse_dir(self, cluster, service):
+ logger.info("Setting the Hive Warehouse Dir")
+ for service in self.api.get_all_clusters()[0].get_all_services():
+ logger.info(service)
+ if service.type == "HIVE":
+ hive_config = { "hive_warehouse_directory" : HIVE_WAREHOUSE_DIR }
+ service.update_config(hive_config)
+
+ # TODO: This functionality should be more generally available to other
infrastructure
+ # code, rather than being quarantined in this script. See IMPALA-4367.
+ @timing
+ def get_service_client_configurations(self):
+ """Download the client configurations necessary to upload data to the
remote
+ cluster. Unfortunately, the CM API does not allow downloading it so we
have to
+ iterate over the services and download the config for all of them.
+
+ In addition, returns an options dictionary with settings required for
data loading
+ like the HS2 server, Impala hosts, Name node etc.
+
+ Returns:
+ A client-configuration dictionary, e.g.:
+
+ {
+ 'hive_warehouse_directory': '/test-warehouse',
+ 'hs2': 'impala-test-cluster-1.gce.cloudera.com:10000',
+ 'impalad': ['impala-test-cluster-4.gce.cloudera.com:21000',
+ 'impala-test-cluster-2.gce.cloudera.com:21000',
+ 'impala-test-cluster-3.gce.cloudera.com:21000'],
+ 'metastore': 'impala-test-cluster-1.gce.cloudera.com:9083',
+ 'namenode': 'impala-test-cluster-1.gce.cloudera.com',
+ 'namenode_http':
'impala-test-cluster-1.gce.cloudera.com:20101',
+ 'kudu_master': 'impala-test-cluster-1.gce.cloudera.com'
+ }
+ """
+ # Iterate overs services and find the information we need
+ result = {}
+ for service_type, service in self.services.iteritems():
+ if service_type == "IMPALA":
+ roles = service.get_roles_by_type("IMPALAD")
+ impalads = []
+ for r in roles:
+ rc_config = r.get_config("full")
+ hostname = self.get_hostname_for_ref(r.hostRef)
+ hs2_port = self.get_or_default(rc_config["beeswax_port"])
+ impalads.append("{0}:{1}".format(hostname, hs2_port))
+ result["impalad"] = impalads
+ elif service_type == "HBASE":
+ self.download_client_config(self.cluster, service)
+ elif service_type == "HDFS":
+ self.download_client_config(self.cluster, service)
+ role = service.get_roles_by_type("NAMENODE")
+ config = role[0].get_config("full")
+ namenode = self.get_hostname_for_ref(role[0].hostRef)
+ result["namenode"] = namenode
+ result["namenode_http"] = "{0}:{1}".format(
+ namenode,
+ self.get_or_default(config["dfs_http_port"])
+ )
+ elif service_type == "HIVE":
+ self.set_hive_warehouse_dir(self.cluster, service)
+ self.download_client_config(self.cluster, service)
+ hs2 = service.get_roles_by_type("HIVESERVER2")[0]
+ rc_config = hs2.get_config("full")
+ result["hive_warehouse_directory"] = self.get_or_default(
+ service.get_config("full")[0]["hive_warehouse_directory"])
+ hostname = self.get_hostname_for_ref(hs2.hostRef)
+ result["hs2"] = "{0}:{1}".format(hostname, self.get_or_default(
+ rc_config["hs2_thrift_address_port"]))
+
+ # Get Metastore information
+ ms = service.get_roles_by_type("HIVEMETASTORE")[0]
+ rc_config = ms.get_config("full")
+ result["metastore"] = "{0}:{1}".format(
+ self.get_hostname_for_ref(ms.hostRef),
+ self.get_or_default(rc_config["hive_metastore_port"])
+ )
+ elif service_type == "KUDU":
+ # Service KUDU does not require a client configuration
+ result["kudu_master"] = self.cm_host
+
+ return result
+
+ # TODO: This functionality should be more generally available to other
infrastructure
+ # code, rather than being quarantined in this script. See IMPALA-4367.
+ @staticmethod
+ def find_snapshot_file(snapshot_dir):
+ """Given snapshot_directory, walks the directory tree until it finds a
file
+ matching the test-warehouse archive pattern."""
+ for root, _, file_names in os.walk(snapshot_dir):
+ for filename in fnmatch.filter(file_names,
"test-warehouse-*-SNAPSHOT.tar.gz"):
+ logger.info("Found Snapshot file {0}".format(filename))
+ return os.path.join(root, filename)
+
+ @timing
+ def prepare(self):
+ """Populate the environment of the process with the necessary values.
+
+ In addition, it creates helper objects to run shell and SSH processes.
+ """
+ # Populate environment with required variables
+ os.environ["HS2_HOST_PORT"] = self.config["hs2"]
+ os.environ["HDFS_NN"] = self.config["namenode"]
+ os.environ["IMPALAD"] = self.config["impalad"][0]
+ os.environ["REMOTE_LOAD"] = "1"
+ os.environ["HADOOP_USER_NAME"] = "hdfs"
+ os.environ["TEST_WAREHOUSE_DIR"] =
self.config["hive_warehouse_directory"]
+ os.environ["KUDU_MASTER"] = self.config["kudu_master"]
+
+ if self.options.snapshot_file is None:
+ if "SNAPSHOT_DIR" in os.environ:
+ snapshot_dir = os.environ["SNAPSHOT_DIR"]
+ else:
+ snapshot_dir = "{0}/testdata/test-warehouse-SNAPSHOT".format(
+ os.getenv("WORKSPACE"))
+ if not os.path.isdir(snapshot_dir):
+ err_msg = 'Snapshot directory "{0}" is not a valid directory'
+ logger.error(err_msg.format(snapshot_dir))
+ raise OSError("Could not find test-warehouse snapshot file.")
+
+ logger.info("Snapshot directory: {0}".format(snapshot_dir))
+ self.snapshot_file = self.find_snapshot_file(snapshot_dir)
+ else:
+ self.snapshot_file = self.options.snapshot_file
+
+ # Prepare shortcuts for connecting to remote services
+ self.gtw_ssh = ssh.bake("{0}@{1}".format(self.options.ssh_user,
self.gateway),
+ "-oStrictHostKeyChecking=no",
+ "-oUserKnownHostsFile=/dev/null",
+ t=True, _out=tee, _err=tee)
+
+ self.beeline = sh.beeline.bake(silent=False, outputformat="csv2",
n="impala",
+ u="jdbc:hive2://{0}/default".format(
+ self.config["hs2"]))
+
+ self.load_test_warehouse = sh.Command(
+ "{0}/testdata/bin/load-test-warehouse-snapshot.sh".format(
+ self.impala_home)).bake(
+ _out=tee, _err=tee)
+
+ self.create_load_data = sh.Command(
+ "{0}/testdata/bin/create-load-data.sh".format(self.impala_home))
+
+ self.main_impalad = self.config["impalad"][0]
+ self.impala_shell =
sh.Command("impala-shell.sh").bake(i=self.main_impalad,
+ _out=tee,
_err=tee)
+
+ self.python = sh.Command("impala-python").bake(u=True)
+ self.compute_stats = sh.Command(
+
"{0}/testdata/bin/compute-table-stats.sh".format(self.impala_home)).bake(
+ _out=tee, _err=tee)
+
+ @timing
+ def load(self):
+ """This method performs the actual data load. First it removes any
known artifacts
+ from the remote location. Next it drops potentially existing database
from the
+ Hive Metastore. Now, it invokes the load-test-warehouse-snapshot.sh and
+ create-load-data.sh scripts with the appropriate parameters. The most
important
+ paramters are implicitly passed to the scripts as environment
variables pointing
+ to the remote HDFS, Hive and Impala.
+ """
+ exploration_strategy = self.options.exploration_strategy
+
+ logger.info("Removing other databases")
+ dblist = self.beeline(e="show databases;", _err=tee).stdout
+ database_list = dblist.split()[1:] # The first element is the header
string
+ for db in database_list:
+ if db.strip() != "default":
+ logger.debug("Dropping database %s", db)
+ self.impala_shell(q="drop database if exists {0}
cascade;".format(db))
+
+ logger.info("Invalidating metadata in Impala")
+ self.impala_shell(q="invalidate metadata;")
+
+ logger.info("Removing previous remote {0}".format(
+ self.config["hive_warehouse_directory"]))
+ r = sh.hdfs.dfs("-rm", "-r", "-f", "{0}".format(
+ self.config["hive_warehouse_directory"]))
+
+ logger.info("Expunging HDFS trash")
+ r = sh.hdfs.dfs("-expunge")
+
+ logger.info("Uploading test warehouse snapshot")
+ self.load_test_warehouse(self.snapshot_file)
+
+ # TODO: We need to confirm that if we change any permissions, that we
don't
+ # affect any running tests. See IMPALA-4375.
+ logger.info("Changing warehouse ownership")
+ r = sh.hdfs.dfs("-chown", "-R", "impala:hdfs", "{0}".format(
+ self.config["hive_warehouse_directory"]))
+ sh.hdfs.dfs("-chmod", "-R", "g+rwx", "{0}".format(
+ self.config["hive_warehouse_directory"]))
+ sh.hdfs.dfs("-chmod", "1777", "{0}".format(
+ self.config["hive_warehouse_directory"]))
+
+ logger.info("Calling create_load_data.sh")
+ # The $USER variable is used in the create-load-data.sh script for
beeline
+ # impersonation.
+ new_env = os.environ.copy()
+ new_env["LOGNAME"] = "impala"
+ new_env["USER"] = "impala"
+ new_env["USERNAME"] = "impala"
+
+ # Regardless of whether we are in fact skipping the snapshot load or
not,
+ # we nonetheless always pass -skip_snapshot_load to
create-load-data.sh.
+ # This is because we have already loaded the snapshot earlier in this
+ # script, so we don't want create-load-data.sh to invoke
+ # load-test-warehouse-snapshot.sh again.
+ #
+ # It would actually be nice to be able to skip the snapshot load, but
+ # because of the existing messiness of create-load-data.sh, we can't.
+ # This invocation...
+ #
+ # $ create-load-data.sh -skip_snapshot_load -exploration_strategy
core
+ #
+ # ...results in this error:
+ #
+ # Creating /test-warehouse HDFS directory \
+ # (logging to create-test-warehouse-dir.log)... FAILED
+ # 'hadoop fs -mkdir /test-warehouse' failed. Tail of log:
+ # Log for command 'hadoop fs -mkdir /test-warehouse'
+ # mkdir: `/test-warehouse': File exists
+ #
+ # Similarly, even though we might pass in "core" as the exploration
strategy,
+ # because we aren't loading a metadata snapshot (i.e.,
-skip_metadata_load is
+ # false), an exhaustive dataload will always be done. This again is
the result
+ # of logic in create-load-data.sh, which itself ignores the value
passed in
+ # for -exploration_strategy.
+ #
+ # See IMPALA-4399: "create-load-data.sh has bitrotted to some extent,
and needs
+ # to be cleaned up"
+ create_load_data_args = ["-skip_snapshot_load", "-cm_host",
self.cm_host,
+ "-snapshot_file", self.snapshot_file,
+ "-exploration_strategy", exploration_strategy]
+
+ self.create_load_data(*create_load_data_args, _env=new_env, _out=tee,
_err=tee)
+
+ sh.hdfs.dfs("-chown", "-R", "impala:hdfs", "{0}".format(
+ self.config["hive_warehouse_directory"]))
+
+ logger.info("Re-load HBase data")
+ # Manually load the HBase data last.
+ self.python("{0}/bin/load-data.py".format(self.impala_home),
+ "--hive_warehouse_dir={0}".format(
+ self.config["hive_warehouse_directory"]),
+ "--table_formats=hbase/none",
+ "--hive_hs2_hostport={0}".format(self.config["hs2"]),
+ "--hdfs_namenode={0}".format(self.config["namenode"]),
+ "--exploration_strategy={0}".format(exploration_strategy),
+ workloads="functional-query",
+ force=True,
+ impalad=self.main_impalad,
+ _env=new_env,
+ _out=tee,
+ _err=tee)
+
+ self.compute_stats()
+ logger.info("Load data finished")
+
+ # TODO: Should this be refactored out of this script? It has nothing to do
with
+ # data loading per se. If tests rely on the environment on the client
being set
+ # a certain way -- as in the prepare() method -- we may need to find
another way
+ # to deal with that. See IMPALA-4376.
+ @timing
+ def test(self):
+ """Execute Impala's end-to-end tests against a remote cluster. All
configuration
+ paramters are picked from the cluster configuration that was fetched
via the
+ CM API."""
+
+ # TODO: Running tests via runtest.py is currently not working against
a remote
+ # cluster (although running directly via py.test seems to work.) This
method
+ # may be refactored out of this file under IMPALA-4376, so for the
time being,
+ # raise a NotImplementedError.
+ raise NotImplementedError
+
+ # Overwrite the username to match the service user on the remote
system and deal
+ # with the assumption that in the local development environment the
current user
+ # is HDFS superuser as well.
+ new_env = os.environ.copy()
+ new_env["LOGNAME"] = "impala"
+ new_env["USER"] = "impala"
+ new_env["USERNAME"] = "impala"
+
+ strategy = self.options.exploration_strategy
+ logger.info("Running tests with exploration strategy
{0}".format(strategy))
+ run_tests =
sh.Command("{0}/tests/run-tests.py".format(self.impala_home))
+ run_tests("--skip_local_tests",
+ "--exploration_strategy={0}".format(strategy),
+
"--workload_exploration_strategy=functional-query:{0}".format(strategy),
+
"--namenode_http_address={0}".format(self.config["namenode_http"]),
+ "--hive_server2={0}".format(self.config["hs2"]),
+ "--metastore_server={0}".format(self.config["metastore"]),
+ "query_test",
+ maxfail=10,
+ impalad=",".join(self.config["impalad"]),
+ _env=new_env,
+ _out=tee,
+ _err=tee)
+
+
+def parse_args():
+ parser = OptionParser()
+ parser.add_option("--snapshot-file", default=None,
+ help="Path to the test-warehouse archive")
+ parser.add_option("--cm-user", default="admin", help="Cloudera Manager
admin user")
+ parser.add_option("--cm-pass", default="admin",
+ help="Cloudera Manager admin user password")
+ parser.add_option("--gateway", default=None,
+ help=("Gateway host to upload the data from. If not set,
uses the "
+ "CM host as gateway."))
+ parser.add_option("--ssh-user", default="jenkins",
+ help=("System user on the remote machine with
passwordless "
+ "SSH configured."))
+ parser.add_option("--no-load", action="store_false", default=True,
dest="load",
+ help="Do not try to load the snapshot")
+ parser.add_option("--exploration-strategy", default="core")
+ parser.add_option("--test", action="store_true", default=False,
+ help="Run end-to-end tests against cluster")
+ parser.set_usage("remote_data_load.py [options] cm_host")
+
+ options, args = parser.parse_args()
+
+ try:
+ return options, args[0] # args[0] is the cm_host
+ except IndexError:
+ logger.error("You must supply the cm_host.")
+ parser.print_usage()
+ raise
+
+def main(cm_host, options):
+ """
+ Load data to a remote cluster (and/or run tests) as specified.
+
+ Args:
+ cm_host: FQDN or IP of the CM host machine
+ options: an optparse 'options' instance containing RemoteDataLoad
+ values (though any object with the correct .attributes, e.g.
+ a collections.namedtuple instance, would also work)
+ """
+ rd = RemoteDataLoad(cm_host, options)
+
+ if options.load:
+ rd.load()
+ if options.test:
+ rd.test()
+
+
+if __name__ == "__main__":
+ options, cm_host = parse_args()
+ main(cm_host=cm_host, options=options)
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ce4c5f67/testdata/bin/compute-table-stats.sh
----------------------------------------------------------------------
diff --git a/testdata/bin/compute-table-stats.sh
b/testdata/bin/compute-table-stats.sh
index 6397d29..f06efab 100755
--- a/testdata/bin/compute-table-stats.sh
+++ b/testdata/bin/compute-table-stats.sh
@@ -23,7 +23,12 @@ set -euo pipefail
trap 'echo Error in $0 at line $LINENO: $(cd "'$PWD'" && awk "NR == $LINENO"
$0)' ERR
. ${IMPALA_HOME}/bin/impala-config.sh > /dev/null 2>&1
-COMPUTE_STATS_SCRIPT="${IMPALA_HOME}/tests/util/compute_table_stats.py"
+
+# TODO: We need a better way of managing how these get set. See:
+# https://issues.cloudera.org/browse/IMPALA-4346
+IMPALAD=${IMPALAD:-localhost:21000}
+
+COMPUTE_STATS_SCRIPT="${IMPALA_HOME}/tests/util/compute_table_stats.py
--impalad=${IMPALAD}"
# Run compute stats over as many of the tables used in the Planner tests as
possible.
${COMPUTE_STATS_SCRIPT} --db_names=functional\
@@ -44,4 +49,3 @@ if "$KUDU_IS_SUPPORTED"; then
${COMPUTE_STATS_SCRIPT} --db_names=functional_kudu
${COMPUTE_STATS_SCRIPT} --db_names=tpch_kudu
fi
-
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ce4c5f67/testdata/bin/create-load-data.sh
----------------------------------------------------------------------
diff --git a/testdata/bin/create-load-data.sh b/testdata/bin/create-load-data.sh
index 7631c1c..7a351fc 100755
--- a/testdata/bin/create-load-data.sh
+++ b/testdata/bin/create-load-data.sh
@@ -34,17 +34,38 @@ trap 'echo Error in $0 at line $LINENO: $(cd "'$PWD'" &&
awk "NR == $LINENO" $0)
. ${IMPALA_HOME}/bin/impala-config.sh > /dev/null 2>&1
. ${IMPALA_HOME}/testdata/bin/run-step.sh
+# Environment variables used to direct the data loading process to an external
cluster.
+# TODO: We need a better way of managing how these get set. See:
+# https://issues.cloudera.org/browse/IMPALA-4346
+: ${HS2_HOST_PORT=localhost:11050}
+: ${HDFS_NN=localhost:20500}
+: ${IMPALAD=localhost:21000}
+: ${REMOTE_LOAD=}
+: ${CM_HOST=}
+
SKIP_METADATA_LOAD=0
SKIP_SNAPSHOT_LOAD=0
SNAPSHOT_FILE=""
LOAD_DATA_ARGS=""
-JDBC_URL="jdbc:hive2://localhost:11050/default;"
+EXPLORATION_STRATEGY="exhaustive"
+export JDBC_URL="jdbc:hive2://${HS2_HOST_PORT}/default;"
+
# For logging when using run-step.
LOG_DIR=${IMPALA_DATA_LOADING_LOGS_DIR}
+echo "Executing: create-load-data.sh $@"
+
while [ -n "$*" ]
do
case $1 in
+ -exploration_strategy)
+ EXPLORATION_STRATEGY=${2-}
+ if [[ -z "$EXPLORATION_STRATEGY" ]]; then
+ echo "Must provide an exploration strategy from e.g. core, exhaustive"
+ exit 1;
+ fi
+ shift;
+ ;;
-skip_metadata_load)
SKIP_METADATA_LOAD=1
;;
@@ -59,11 +80,16 @@ do
fi
shift;
;;
+ -cm_host)
+ CM_HOST=${2-}
+ shift;
+ ;;
-help|-h|*)
echo "create-load-data.sh : Creates data and loads from scratch"
echo "[-skip_metadata_load] : Skips loading of metadata"
echo "[-skip_snapshot_load] : Assumes that the snapshot is already
loaded"
echo "[-snapshot_file] : Loads the test warehouse snapshot into hdfs"
+ echo "[-cm_host] : Address of the Cloudera Manager host if loading to a
remote cluster"
exit 1;
;;
esac
@@ -71,8 +97,10 @@ do
done
if [[ $SKIP_METADATA_LOAD -eq 0 && "$SNAPSHOT_FILE" = "" ]]; then
- run-step "Loading Hive Builtins" load-hive-builtins.log \
+ if [[ -z "$REMOTE_LOAD" ]]; then
+ run-step "Loading Hive Builtins" load-hive-builtins.log \
${IMPALA_HOME}/testdata/bin/load-hive-builtins.sh
+ fi
run-step "Generating HBase data" create-hbase.log \
${IMPALA_HOME}/testdata/bin/create-hbase.sh
run-step "Creating /test-warehouse HDFS directory"
create-test-warehouse-dir.log \
@@ -98,6 +126,14 @@ else
echo Skipping loading data to hdfs.
fi
+echo "Derived params for create-load-data.sh:"
+echo "EXPLORATION_STRATEGY=${EXPLORATION_STRATEGY:-}"
+echo "SKIP_METADATA_LOAD=${SKIP_METADATA_LOAD:-}"
+echo "SKIP_SNAPSHOT_LOAD=${SKIP_SNAPSHOT_LOAD:-}"
+echo "SNAPSHOT_FILE=${SNAPSHOT_FILE:-}"
+echo "CM_HOST=${CM_HOST:-}"
+echo "REMOTE_LOAD=${REMOTE_LOAD:-}"
+
function load-custom-schemas {
SCHEMA_SRC_DIR=${IMPALA_HOME}/testdata/data/schemas
SCHEMA_DEST_DIR=/test-warehouse/schemas
@@ -131,7 +167,7 @@ function load-data {
MSG="Loading workload '$WORKLOAD'"
ARGS=("--workloads $WORKLOAD")
- MSG+=" Using exploration strategy '$EXPLORATION_STRATEGY'"
+ MSG+=" using exploration strategy '$EXPLORATION_STRATEGY'"
ARGS+=("-e $EXPLORATION_STRATEGY")
if [ $TABLE_FORMATS ]; then
MSG+=" in table formats '$TABLE_FORMATS'"
@@ -144,15 +180,35 @@ function load-data {
if [ "${WORKLOAD}" = "functional-query" ]; then
WORKLOAD="functional"
fi
+
+ # TODO: Why is there a REMOTE_LOAD condition?
+ # See https://issues.cloudera.org/browse/IMPALA-4347
+ #
# Force load the dataset if we detect a schema change.
- if ! ${IMPALA_HOME}/testdata/bin/check-schema-diff.sh $WORKLOAD; then
- ARGS+=("--force")
- echo "Force loading $WORKLOAD because a schema change was detected"
- elif [ "${FORCE_LOAD}" = "force" ]; then
- ARGS+=("--force")
- echo "Force loading."
+ if [[ -z "$REMOTE_LOAD" ]]; then
+ if ! ${IMPALA_HOME}/testdata/bin/check-schema-diff.sh $WORKLOAD; then
+ ARGS+=("--force")
+ echo "Force loading $WORKLOAD because a schema change was detected"
+ elif [ "${FORCE_LOAD}" = "force" ]; then
+ ARGS+=("--force")
+ echo "Force loading."
+ fi
fi
-
LOG_FILE=${IMPALA_DATA_LOADING_LOGS_DIR}/data-load-${WORKLOAD}-${EXPLORATION_STRATEGY}.log
+
+ ARGS+=("--impalad ${IMPALAD}")
+ ARGS+=("--hive_hs2_hostport ${HS2_HOST_PORT}")
+ ARGS+=("--hdfs_namenode ${HDFS_NN}")
+
+ if [[ -n ${TABLE_FORMATS} ]]; then
+ # TBL_FMT_STR replaces slashes with underscores,
+ # e.g., kudu/none/none -> kudu_none_none
+ TBL_FMT_STR=${TABLE_FORMATS//[\/]/_}
+
LOG_BASENAME=data-load-${WORKLOAD}-${EXPLORATION_STRATEGY}-${TBL_FMT_STR}.log
+ else
+ LOG_BASENAME=data-load-${WORKLOAD}-${EXPLORATION_STRATEGY}.log
+ fi
+
+ LOG_FILE=${IMPALA_DATA_LOADING_LOGS_DIR}/${LOG_BASENAME}
echo "$MSG. Logging to ${LOG_FILE}"
# Use unbuffered logging by executing with -u
if ! impala-python -u ${IMPALA_HOME}/bin/load-data.py ${ARGS[@]} &>
${LOG_FILE}; then
@@ -165,10 +221,13 @@ function load-data {
function cache-test-tables {
echo CACHING tpch.nation AND functional.alltypestiny
# uncaching the tables first makes this operation idempotent.
- ${IMPALA_HOME}/bin/impala-shell.sh -q "alter table functional.alltypestiny
set uncached"
- ${IMPALA_HOME}/bin/impala-shell.sh -q "alter table tpch.nation set uncached"
- ${IMPALA_HOME}/bin/impala-shell.sh -q "alter table tpch.nation set cached in
'testPool'"
- ${IMPALA_HOME}/bin/impala-shell.sh -q\
+ ${IMPALA_HOME}/bin/impala-shell.sh -i ${IMPALAD}\
+ -q "alter table functional.alltypestiny set uncached"
+ ${IMPALA_HOME}/bin/impala-shell.sh -i ${IMPALAD}\
+ -q "alter table tpch.nation set uncached"
+ ${IMPALA_HOME}/bin/impala-shell.sh -i ${IMPALAD}\
+ -q "alter table tpch.nation set cached in 'testPool'"
+ ${IMPALA_HOME}/bin/impala-shell.sh -i ${IMPALAD} -q\
"alter table functional.alltypestiny set cached in 'testPool'"
}
@@ -179,6 +238,9 @@ function load-aux-workloads {
if [ -d ${IMPALA_AUX_WORKLOAD_DIR} ] && [ -d ${IMPALA_AUX_DATASET_DIR} ];
then
echo Loading auxiliary workloads. Logging to $LOG_FILE.
if ! impala-python -u ${IMPALA_HOME}/bin/load-data.py --workloads all\
+ --impalad=${IMPALAD}\
+ --hive_hs2_hostport=${HS2_HOST_PORT}\
+ --hdfs_namenode=${HDFS_NN}\
--workload_dir=${IMPALA_AUX_WORKLOAD_DIR}\
--dataset_dir=${IMPALA_AUX_DATASET_DIR}\
--exploration_strategy=core ${LOAD_DATA_ARGS} >> $LOG_FILE 2>&1; then
@@ -192,6 +254,7 @@ function load-aux-workloads {
}
function copy-auth-policy {
+ echo COPYING AUTHORIZATION POLICY FILE
hadoop fs -rm -f ${FILESYSTEM_PREFIX}/test-warehouse/authz-policy.ini
hadoop fs -put ${IMPALA_HOME}/fe/src/test/resources/authz-policy.ini \
${FILESYSTEM_PREFIX}/test-warehouse/
@@ -207,18 +270,37 @@ function copy-and-load-dependent-tables {
hadoop fs -rm -r -f /tmp/alltypes_seq
hadoop fs -mkdir -p /tmp/alltypes_seq/year=2009
hadoop fs -mkdir -p /tmp/alltypes_rc/year=2009
- hadoop fs -cp /test-warehouse/alltypes_seq/year=2009/month=2/
/tmp/alltypes_seq/year=2009
- hadoop fs -cp /test-warehouse/alltypes_rc/year=2009/month=3/
/tmp/alltypes_rc/year=2009
+ hadoop fs -cp /test-warehouse/alltypes_seq/year=2009/month=2/
/tmp/alltypes_seq/year=2009
+ hadoop fs -cp /test-warehouse/alltypes_rc/year=2009/month=3/
/tmp/alltypes_rc/year=2009
# Create a hidden file in AllTypesSmall
hadoop fs -rm -f /test-warehouse/alltypessmall/year=2009/month=1/_hidden
hadoop fs -rm -f /test-warehouse/alltypessmall/year=2009/month=1/.hidden
- hadoop fs -cp
/test-warehouse/zipcode_incomes/DEC_00_SF3_P077_with_ann_noheader.csv \
+ hadoop fs -cp
/test-warehouse/zipcode_incomes/DEC_00_SF3_P077_with_ann_noheader.csv \
/test-warehouse/alltypessmall/year=2009/month=1/_hidden
- hadoop fs -cp
/test-warehouse/zipcode_incomes/DEC_00_SF3_P077_with_ann_noheader.csv \
+ hadoop fs -cp
/test-warehouse/zipcode_incomes/DEC_00_SF3_P077_with_ann_noheader.csv \
/test-warehouse/alltypessmall/year=2009/month=1/.hidden
- # For tables that rely on loading data from local fs test-warehouse
+ # In case the data is updated by a non-super user, make sure the user can
write
+ # by chmoding 777 /tmp/alltypes_rc and /tmp/alltypes_seq. This is needed in
order
+ # to prevent this error during data load to a remote cluster:
+ #
+ # ERROR : Failed with exception Unable to move source
hdfs://cluster-1.foo.cloudera.com:
+ # 8020/tmp/alltypes_seq/year=2009/month=2/000023_0 to destination
hdfs://cluster-1.foo.
+ #
cloudera.com:8020/test-warehouse/alltypesmixedformat/year=2009/month=2/000023_0
+ # [...]
+ # Caused by: org.apache.hadoop.security.AccessControlException:
+ # Permission denied: user=impala, access=WRITE
+ # inode="/tmp/alltypes_seq/year=2009/month=2":hdfs:supergroup:drwxr-xr-x
+ #
+ # The error occurs while loading dependent tables.
+ #
+ # See: logs/data_loading/copy-and-load-dependent-tables.log)
+ # See also: https://issues.cloudera.org/browse/IMPALA-4345
+ hadoop fs -chmod -R 777 /tmp/alltypes_rc
+ hadoop fs -chmod -R 777 /tmp/alltypes_seq
+
+ # For tables that rely on loading data from local fs test-wareload-house
# TODO: Find a good way to integrate this with the normal data loading
scripts
beeline -n $USER -u "${JDBC_URL}" -f\
${IMPALA_HOME}/testdata/bin/load-dependent-tables.sql
@@ -249,6 +331,7 @@ EOF
function load-custom-data {
# Load the index files for corrupted lzo data.
+ hadoop fs -mkdir -p /test-warehouse/bad_text_lzo_text_lzo
hadoop fs -rm -f /test-warehouse/bad_text_lzo_text_lzo/bad_text.lzo.index
hadoop fs -put ${IMPALA_HOME}/testdata/bad_text_lzo/bad_text.lzo.index \
/test-warehouse/bad_text_lzo_text_lzo/
@@ -258,8 +341,12 @@ function load-custom-data {
# Cleanup the old bad_text_lzo files, if they exist.
hadoop fs -rm -r -f /test-warehouse/bad_text_lzo/
- # Index all lzo files in HDFS under /test-warehouse
- ${IMPALA_HOME}/testdata/bin/lzo_indexer.sh /test-warehouse
+ # TODO: Why is there a REMOTE_LOAD condition?
+ # See https://issues.cloudera.org/browse/IMPALA-4347
+ if [[ -z $REMOTE_LOAD ]]; then
+ # Index all lzo files in HDFS under /test-warehouse
+ ${IMPALA_HOME}/testdata/bin/lzo_indexer.sh /test-warehouse
+ fi
hadoop fs -mv /bad_text_lzo_text_lzo/ /test-warehouse/
@@ -292,7 +379,7 @@ function load-custom-data {
done
# Remove all index files in this partition.
- hadoop fs -rm /test-warehouse/alltypes_text_lzo/year=2009/month=1/*.lzo.index
+ hadoop fs -rm -f
/test-warehouse/alltypes_text_lzo/year=2009/month=1/*.lzo.index
# Add a sequence file that only contains a header (see IMPALA-362)
hadoop fs -put -f
${IMPALA_HOME}/testdata/tinytable_seq_snap/tinytable_seq_snap_header_only \
@@ -303,8 +390,16 @@ function load-custom-data {
hadoop fs -put -f
${IMPALA_HOME}/testdata/compressed_formats/compressed_payload.snap \
/test-warehouse/compressed_payload.snap
+ # Create Avro tables
beeline -n $USER -u "${JDBC_URL}" -f\
${IMPALA_HOME}/testdata/avro_schema_resolution/create_table.sql
+
+ # Delete potentially existing avro data
+ hadoop fs -rm -f /test-warehouse/avro_schema_resolution_test/*.avro
+
+ # Upload Avro data to the 'schema_resolution_test' table
+ hadoop fs -put ${IMPALA_HOME}/testdata/avro_schema_resolution/records*.avro \
+ /test-warehouse/avro_schema_resolution_test
}
function build-and-copy-hive-udfs {
@@ -319,12 +414,16 @@ function build-and-copy-hive-udfs {
# Additional data loading actions that must be executed after the main data is
loaded.
function custom-post-load-steps {
- # Configure alltypes_seq as a read-only table. This is required for fe tests.
- # Set both read and execute permissions because accessing the contents of a
directory on
- # the local filesystem requires the x permission (while on HDFS it requires
the r
- # permission).
- hadoop fs -chmod -R 555
${FILESYSTEM_PREFIX}/test-warehouse/alltypes_seq/year=2009/month=1
- hadoop fs -chmod -R 555
${FILESYSTEM_PREFIX}/test-warehouse/alltypes_seq/year=2009/month=3
+ # TODO: Why is there a REMOTE_LOAD condition?
+ # See https://issues.cloudera.org/browse/IMPALA-4347
+ if [[ -z "$REMOTE_LOAD" ]]; then
+ # Configure alltypes_seq as a read-only table. This is required for fe
tests.
+ # Set both read and execute permissions because accessing the contents of
a directory on
+ # the local filesystem requires the x permission (while on HDFS it
requires the r
+ # permission).
+ hadoop fs -chmod -R 555
${FILESYSTEM_PREFIX}/test-warehouse/alltypes_seq/year=2009/month=1
+ hadoop fs -chmod -R 555
${FILESYSTEM_PREFIX}/test-warehouse/alltypes_seq/year=2009/month=3
+ fi
#IMPALA-1881: data file produced by hive with multiple blocks.
hadoop fs -mkdir -p
${FILESYSTEM_PREFIX}/test-warehouse/lineitem_multiblock_parquet
@@ -349,7 +448,7 @@ function copy-and-load-ext-data-source {
# Copy the test data source library into HDFS
${IMPALA_HOME}/testdata/bin/copy-data-sources.sh
# Create data sources table.
- ${IMPALA_HOME}/bin/impala-shell.sh -f\
+ ${IMPALA_HOME}/bin/impala-shell.sh -i ${IMPALAD} -f\
${IMPALA_HOME}/testdata/bin/create-data-source-table.sql
}
@@ -365,9 +464,12 @@ if [[ "${TARGET_FILESYSTEM}" == "local" ]]; then
else
START_CLUSTER_ARGS="-s 3 ${START_CLUSTER_ARGS}"
fi
-run-step "Starting Impala cluster" start-impala-cluster.log \
- ${IMPALA_HOME}/bin/start-impala-cluster.py \
- --log_dir=${IMPALA_DATA_LOADING_LOGS_DIR} ${START_CLUSTER_ARGS}
+if [[ -z "$REMOTE_LOAD" ]]; then
+ run-step "Starting Impala cluster" start-impala-cluster.log \
+ ${IMPALA_HOME}/bin/start-impala-cluster.py
--log_dir=${IMPALA_DATA_LOADING_LOGS_DIR} \
+ ${START_CLUSTER_ARGS}
+fi
+
# The hdfs environment script sets up kms (encryption) and cache pools (hdfs
caching).
# On a non-hdfs filesystem, we don't test encryption or hdfs caching, so this
setup is not
# needed.
@@ -383,8 +485,11 @@ if [ $SKIP_METADATA_LOAD -eq 0 ]; then
run-step "Loading TPC-H data" load-tpch.log load-data "tpch" "core"
# Load tpch nested data.
# TODO: Hacky and introduces more complexity into the system, but it is
expedient.
+ if [[ -n "$CM_HOST" ]]; then
+ LOAD_NESTED_ARGS="--cm-host $CM_HOST"
+ fi
run-step "Loading nested data" load-nested.log \
- ${IMPALA_HOME}/testdata/bin/load_nested.py
+ ${IMPALA_HOME}/testdata/bin/load_nested.py ${LOAD_NESTED_ARGS:-}
run-step "Loading TPC-DS data" load-tpcds.log load-data "tpcds" "core"
run-step "Loading auxiliary workloads" load-aux-workloads.log
load-aux-workloads
run-step "Loading dependent tables" copy-and-load-dependent-tables.log \
@@ -419,7 +524,10 @@ if [ "${TARGET_FILESYSTEM}" = "hdfs" ]; then
run-step "Loading external data sources" load-ext-data-source.log \
copy-and-load-ext-data-source
- run-step "Splitting HBase" create-hbase.log
${IMPALA_HOME}/testdata/bin/split-hbase.sh
+ # HBase splitting is only relevant for FE tests
+ if [[ -z "$REMOTE_LOAD" ]]; then
+ run-step "Splitting HBase" create-hbase.log
${IMPALA_HOME}/testdata/bin/split-hbase.sh
+ fi
run-step "Creating internal HBase table" create-internal-hbase-table.log \
create-internal-hbase-table
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ce4c5f67/testdata/bin/create-table-many-blocks.sh
----------------------------------------------------------------------
diff --git a/testdata/bin/create-table-many-blocks.sh
b/testdata/bin/create-table-many-blocks.sh
index 5b0199a..abdf94f 100755
--- a/testdata/bin/create-table-many-blocks.sh
+++ b/testdata/bin/create-table-many-blocks.sh
@@ -29,6 +29,12 @@ trap 'echo Error in $0 at line $LINENO: $(cd "'$PWD'" && awk
"NR == $LINENO" $0)
. ${IMPALA_HOME}/bin/impala-config.sh > /dev/null 2>&1
+# Environment variables needed for remote cluster
+: ${HS2_HOST_PORT:=localhost:11050}
+JDBC_URL="jdbc:hive2://${HS2_HOST_PORT}/default;"
+
+HIVE_CMD="beeline -n $USER -u $JDBC_URL"
+
LOCAL_OUTPUT_DIR=$(mktemp -dt "impala_test_tmp.XXXXXX")
echo $LOCAL_OUTPUT_DIR
@@ -66,9 +72,9 @@
HDFS_PATH=/test-warehouse/many_blocks_num_blocks_per_partition_${BLOCKS_PER_PART
DB_NAME=scale_db
TBL_NAME=num_partitions_${NUM_PARTITIONS}_blocks_per_partition_${BLOCKS_PER_PARTITION}
-hive -e "create database if not exists scale_db"
-hive -e "drop table if exists ${DB_NAME}.${TBL_NAME}"
-hive -e "create external table ${DB_NAME}.${TBL_NAME} (i int) partitioned by
(j int)"
+$HIVE_CMD -e "create database if not exists scale_db"
+$HIVE_CMD -e "drop table if exists ${DB_NAME}.${TBL_NAME}"
+$HIVE_CMD -e "create external table ${DB_NAME}.${TBL_NAME} (i int) partitioned
by (j int)"
# Generate many (small) files. Each file will be assigned a unique block.
echo "Generating ${BLOCKS_PER_PARTITION} files"
@@ -98,6 +104,6 @@ done
echo ";" >> ${LOCAL_OUTPUT_DIR}/hive_create_partitions.q
echo "Executing DDL via Hive"
-hive -f ${LOCAL_OUTPUT_DIR}/hive_create_partitions.q
+$HIVE_CMD -f ${LOCAL_OUTPUT_DIR}/hive_create_partitions.q
echo "Done! Final result in table: ${DB_NAME}.${TBL_NAME}"
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ce4c5f67/testdata/bin/generate-schema-statements.py
----------------------------------------------------------------------
diff --git a/testdata/bin/generate-schema-statements.py
b/testdata/bin/generate-schema-statements.py
index 4c8ef02..1fe6a01 100755
--- a/testdata/bin/generate-schema-statements.py
+++ b/testdata/bin/generate-schema-statements.py
@@ -408,10 +408,11 @@ def build_load_statement(load_template, db_name,
db_suffix, table_name):
db_name=db_name,
db_suffix=db_suffix)
else:
+ base_load_dir = os.getenv("REMOTE_LOAD", os.getenv("IMPALA_HOME"))
load_template = load_template.format(table_name=table_name,
db_name=db_name,
db_suffix=db_suffix,
- impala_home =
os.environ['IMPALA_HOME'])
+ impala_home = base_load_dir)
return load_template
def build_hbase_create_stmt(db_name, table_name, column_families):
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ce4c5f67/testdata/bin/load-test-warehouse-snapshot.sh
----------------------------------------------------------------------
diff --git a/testdata/bin/load-test-warehouse-snapshot.sh
b/testdata/bin/load-test-warehouse-snapshot.sh
index 2b6e90d..604c8f1 100755
--- a/testdata/bin/load-test-warehouse-snapshot.sh
+++ b/testdata/bin/load-test-warehouse-snapshot.sh
@@ -28,13 +28,14 @@ set -euo pipefail
trap 'echo Error in $0 at line $LINENO: $(cd "'$PWD'" && awk "NR == $LINENO"
$0)' ERR
. ${IMPALA_HOME}/bin/impala-config.sh > /dev/null 2>&1
+: ${REMOTE_LOAD:=}
if [[ $# -ne 1 ]]; then
echo "Usage: load-test-warehouse-snapshot.sh
[test-warehouse-SNAPSHOT.tar.gz]"
exit 1
fi
-TEST_WAREHOUSE_DIR="/test-warehouse"
+: ${TEST_WAREHOUSE_DIR=/test-warehouse}
SNAPSHOT_FILE=$1
if [ ! -f ${SNAPSHOT_FILE} ]; then
@@ -42,9 +43,13 @@ if [ ! -f ${SNAPSHOT_FILE} ]; then
exit 1
fi
-echo "Your existing ${TARGET_FILESYSTEM} warehouse directory " \
- "(${FILESYSTEM_PREFIX}${TEST_WAREHOUSE_DIR}) will be removed."
-read -p "Continue (y/n)? "
+if [[ -z "$REMOTE_LOAD" ]]; then
+ echo "Your existing ${TARGET_FILESYSTEM} warehouse directory " \
+ "(${FILESYSTEM_PREFIX}${TEST_WAREHOUSE_DIR} will be removed."
+ read -p "Continue (y/n)? "
+else
+ REPLY=y
+fi
if [[ "$REPLY" =~ ^[Yy]$ ]]; then
# Create a new warehouse directory. If one already exist, remove it first.
if [ "${TARGET_FILESYSTEM}" = "s3" ]; then
@@ -56,7 +61,7 @@ if [[ "$REPLY" =~ ^[Yy]$ ]]; then
else
# Either isilon or hdfs, no change in procedure.
if hadoop fs -test -d ${FILESYSTEM_PREFIX}${TEST_WAREHOUSE_DIR}; then
- echo "Removing existing test-warehouse directory"
+ echo "Removing existing ${TEST_WAREHOUSE_DIR} directory"
# For filesystems that don't allow 'rm' without 'x', chmod to 777 for the
# subsequent 'rm -r'.
if [ "${TARGET_FILESYSTEM}" = "isilon" ] || \
@@ -65,8 +70,13 @@ if [[ "$REPLY" =~ ^[Yy]$ ]]; then
fi
hadoop fs -rm -r -skipTrash ${FILESYSTEM_PREFIX}${TEST_WAREHOUSE_DIR}
fi
- echo "Creating test-warehouse directory"
- hadoop fs -mkdir ${FILESYSTEM_PREFIX}${TEST_WAREHOUSE_DIR}
+ echo "Creating ${TEST_WAREHOUSE_DIR} directory"
+ hadoop fs -mkdir -p ${FILESYSTEM_PREFIX}${TEST_WAREHOUSE_DIR}
+
+ # TODO: commented out because of regressions in local end-to-end testing
+ # See: https://issues.cloudera.org/browse/IMPALA-4345
+ #
+ # hdfs dfs -chmod 1777 ${FILESYSTEM_PREFIX}${TEST_WAREHOUSE_DIR}
fi
else
echo -e "\nAborting."
@@ -81,14 +91,18 @@ mkdir ${SNAPSHOT_STAGING_DIR}
echo "Extracting tarball"
tar -C ${SNAPSHOT_STAGING_DIR} -xzf ${SNAPSHOT_FILE}
-if [ ! -f ${SNAPSHOT_STAGING_DIR}/test-warehouse/githash.txt ]; then
+if [ ! -f ${SNAPSHOT_STAGING_DIR}${TEST_WAREHOUSE_DIR}/githash.txt ]; then
echo "The test-warehouse snapshot does not contain a githash.txt file,
aborting load"
exit 1
fi
-echo "Loading hive builtins"
-${IMPALA_HOME}/testdata/bin/load-hive-builtins.sh
+# Hive builtins are already present on a pre-setup CM managed cluster.
+if [[ -z "$REMOTE_LOAD" ]]; then
+ echo "Loading hive builtins"
+ ${IMPALA_HOME}/testdata/bin/load-hive-builtins.sh
+fi
+
echo "Copying data to ${TARGET_FILESYSTEM}"
if [ "${TARGET_FILESYSTEM}" = "s3" ]; then
# hive does not yet work well with s3, so we won't need hive builtins.
@@ -99,8 +113,7 @@ if [ "${TARGET_FILESYSTEM}" = "s3" ]; then
exit 1
fi
else
- hadoop fs -put ${SNAPSHOT_STAGING_DIR}${TEST_WAREHOUSE_DIR}/* \
- ${FILESYSTEM_PREFIX}${TEST_WAREHOUSE_DIR}
+ hadoop fs -put ${SNAPSHOT_STAGING_DIR}${TEST_WAREHOUSE_DIR}/*
${FILESYSTEM_PREFIX}${TEST_WAREHOUSE_DIR}
fi
${IMPALA_HOME}/bin/create_testdata.sh
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ce4c5f67/testdata/bin/load_nested.py
----------------------------------------------------------------------
diff --git a/testdata/bin/load_nested.py b/testdata/bin/load_nested.py
index f68f8c4..b44bd04 100755
--- a/testdata/bin/load_nested.py
+++ b/testdata/bin/load_nested.py
@@ -20,10 +20,13 @@
'''This script creates a nested version of TPC-H. Non-nested TPC-H must
already be
loaded.
'''
-
import logging
import os
+from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser
+import tests.comparison.cli_options as cli_options
+
+
LOG = logging.getLogger(os.path.splitext(os.path.basename(__file__))[0])
# These vars are set after arg parsing.
@@ -281,23 +284,25 @@ def load():
DROP TABLE tmp_supplier_string;""".split(";"):
if not stmt.strip():
continue
+ LOG.info("Executing: {0}".format(stmt))
hive.execute(stmt)
with cluster.impala.cursor(db_name=target_db) as impala:
impala.invalidate_metadata()
impala.compute_stats()
+ LOG.info("Done loading nested TPCH data")
if __name__ == "__main__":
- from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser
- import tests.comparison.cli_options as cli_options
parser = ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter)
cli_options.add_logging_options(parser)
- cli_options.add_cluster_options(parser)
+ cli_options.add_cluster_options(parser) # --cm-host and similar args added
here
+
parser.add_argument("-s", "--source-db", default="tpch_parquet")
parser.add_argument("-t", "--target-db", default="tpch_nested_parquet")
parser.add_argument("-c", "-p", "--chunks", type=int, default=1)
+
args = parser.parse_args()
cli_options.configure_logging(args.log_level,
debug_log_file=args.debug_log_file)
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ce4c5f67/testdata/bin/run-step.sh
----------------------------------------------------------------------
diff --git a/testdata/bin/run-step.sh b/testdata/bin/run-step.sh
index faef360..e3e516e 100755
--- a/testdata/bin/run-step.sh
+++ b/testdata/bin/run-step.sh
@@ -24,6 +24,7 @@
# outputs if there is an error.
# Usage: run-step <step description> <log file name> <cmd> <arg1> <arg2> ...
# LOG_DIR must be set to a writable directory for logs.
+
function run-step {
local MSG=$1
shift
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ce4c5f67/testdata/bin/setup-hdfs-env.sh
----------------------------------------------------------------------
diff --git a/testdata/bin/setup-hdfs-env.sh b/testdata/bin/setup-hdfs-env.sh
index 921ec67..259cac1 100755
--- a/testdata/bin/setup-hdfs-env.sh
+++ b/testdata/bin/setup-hdfs-env.sh
@@ -16,10 +16,12 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
+#
set -euo pipefail
trap 'echo Error in $0 at line $LINENO: $(cd "'$PWD'" && awk "NR == $LINENO"
$0)' ERR
+: ${REMOTE_LOAD:=}
+
# Create a cache pool and encryption keys for tests
PREVIOUS_PRINCIPAL=""
@@ -27,13 +29,23 @@ CACHEADMIN_ARGS=""
# If we're kerberized, we need to become hdfs for this:
if ${CLUSTER_DIR}/admin is_kerberized; then
- PREVIOUS_PRINCIPAL=`klist | grep ^Default | awk '{print $3}'`
- PREVIOUS_USER=`echo ${PREVIOUS_PRINCIPAL} | awk -F/ '{print $1}'`
- CACHEADMIN_ARGS="-group supergroup -owner ${PREVIOUS_USER}"
- kinit -k -t ${KRB5_KTNAME} ${MINIKDC_PRINC_HDFS}
+ if [[ -n "${REMOTE_LOAD:-}" ]]; then
+ echo "REMOTE_LOAD: $REMOTE_LOAD"
+ echo "Remote cluster testing is not supported with Kerberos"
+ exit 1
+ else
+ PREVIOUS_PRINCIPAL=`klist | grep ^Default | awk '{print $3}'`
+ PREVIOUS_USER=`echo ${PREVIOUS_PRINCIPAL} | awk -F/ '{print $1}'`
+ CACHEADMIN_ARGS="-group supergroup -owner ${PREVIOUS_USER}"
+ kinit -k -t ${KRB5_KTNAME} ${MINIKDC_PRINC_HDFS}
+ fi
fi
-if [[ $TARGET_FILESYSTEM == hdfs ]]; then # Otherwise assume KMS isn't setup.
+# TODO: Investigate how to setup encryption keys for running HDFS encryption
tests
+# against a remote cluster, rather than the local mini-cluster (i.e., when
REMOTE_LOAD
+# is true. See: https://issues.cloudera.org/browse/IMPALA-4344)
+
+if [[ $TARGET_FILESYSTEM == hdfs && -z "$REMOTE_LOAD" ]]; then # Otherwise
assume KMS isn't setup.
# Create encryption keys for HDFS encryption tests. Keys are stored by the
KMS.
EXISTING_KEYS=$(hadoop key list)
for KEY in testkey{1,2}; do
@@ -44,7 +56,13 @@ if [[ $TARGET_FILESYSTEM == hdfs ]]; then # Otherwise
assume KMS isn't setup.
done
fi
-# Create test cache pool
+if [[ -n "${REMOTE_LOAD:-}" ]]; then
+ # Create test cache pool if HADOOP_USER_NAME has a non-zero length
+ if [[ -n "${HADOOP_USER_NAME:-}" ]]; then
+ CACHEADMIN_ARGS="${CACHEADMIN_ARGS} -owner ${USER}"
+ fi
+fi
+
if hdfs cacheadmin -listPools testPool | grep testPool &>/dev/null; then
hdfs cacheadmin -removePool testPool
fi
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ce4c5f67/testdata/datasets/functional/schema_constraints.csv
----------------------------------------------------------------------
diff --git a/testdata/datasets/functional/schema_constraints.csv
b/testdata/datasets/functional/schema_constraints.csv
index d6d1111..790dba4 100644
--- a/testdata/datasets/functional/schema_constraints.csv
+++ b/testdata/datasets/functional/schema_constraints.csv
@@ -118,6 +118,11 @@ table_name:overflow, constraint:restrict_to,
table_format:text/none/none
# seem to like this.
table_name:widerow, constraint:exclude, table_format:hbase/none/none
+# Wide tables fail due to the SERDEPROPERTIES limits. See HIVE-1364.
+table_name:widetable_250_cols, constraints:exclude,
table_format:hbase/none/none
+table_name:widetable_500_cols, constraints:exclude,
table_format:hbase/none/none
+table_name:widetable_1000_cols, constraints:exclude,
table_format:hbase/none/none
+
# nullformat_custom is used in null-insert tests, which user insert overwrite,
# which is not supported in hbase. The schema is also specified in HIVE_CREATE
# with no corresponding LOAD statement.
