Hi Casey,
Data loading issues on ppc are resolved. I have been able to successfully
complete the data loading on ppc for Impala. The FE tests are also test
successfully with 545 tests passing and 36 tests skipped.
I also executed the Custom cluster tests, (tests=41, failures=5, errors=0,
skipped=0). PFA the log for the same. (See attached file:
8June_cc_tests.txt)
It would be great if you could share any pointers on these issues.
Regards,
Valencia
From: Casey Ching <[email protected]>
To: Alex Behm <[email protected]>, Valencia
Serrao/Austin/Contr/IBM@IBMUS, [email protected]
Cc: Valencia Serrao/Austin/Contr/IBM@IBMUS,
[email protected], David
Clissold/Austin/IBM@IBMUS, Sudarshan
Jagadale/Austin/Contr/IBM@IBMUS, Nishidha
Panpaliya/Austin/Contr/IBM@IBMUS
Date: 05/09/2016 10:45 PM
Subject: Re: Fw: Issues with generating testdata for Impala
Hi Valencia,
Have you tried setting up an x86 environment? That could be useful for
comparing to the ppc environment to see what is/isn’t working and being
able to see what the logs should look like.
If the tpch database isn’t there, that should mean data loading failed and
there should have been an error that caused the data loading to exit early
along with an error message in the logs. Did you see anything like that?
You might want to try only running the data loading step, then verifying
that the tpch database exists afterwards.
Casey
On May 9, 2016 at 5:27:49 AM, Valencia Serrao ([email protected]) wrote:
Hi Alex/Casey,
I re-ran the fe tests with the testdata you provided, but the result
is the same as that reported in the earlier mail, with most of the
failures occurring due to tpch database not existing.
Steps followed to test are as follows:
1. copy the testdata to IMPALA_HOME/testdata/impala-data.
2. ./buildall.sh -notests -noclean -format -testdata
3. ./bin/run_all_tests.sh
We had also tried the testdata generation on Ubuntu x86 ppc machine
however, it stops at the same "Invalidate Metadata" step with the
exception.
Any pointers on these issues will be helpful.
Regards,
Valencia
Valencia Serrao---05/05/2016 06:47:59 PM---Hi Alex/Casey, I tried to
run the frontend tests with the data provided. Following is the
result:
From: Valencia Serrao/Austin/Contr/IBM
To: Casey Ching <[email protected]>
Cc: Alex Behm <[email protected]>,
[email protected], Nishidha
Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan
Jagadale/Austin/Contr/IBM@IBMUS, David Clissold/Austin/IBM@IBMUS,
Valencia Serrao/Austin/Contr/IBM@IBMUS
Date: 05/05/2016 06:47 PM
Subject: Re: Fw: Issues with generating testdata for Impala
Hi Alex/Casey,
I tried to run the frontend tests with the data provided. Following
is the result:
Tests run: 545, Failures: 226, Errors: 77, Skipped: 36 [attachment
"data-load-functional-exhaustive.zip" deleted by Valencia
Serrao/Austin/Contr/IBM]
Earlier, the number of "Errors" were 87 , so now they have reduced by
10. However, the "Failures" count is still the same. Most of the
Failures in PlannerTest and AuthorizationTest are related to tpch
(e.g. Database doesn't exist: tpch).
With regard to the directory "impala_data", i've observed that it is
not being accessed/used by any script. Are we missing on any
configuration ?
Kindly guide me on this.
Regards,
Valencia
Valencia Serrao---05/05/2016 02:21:56 PM---Thanks, Casey! I will let
you know the test status.
From: Valencia Serrao/Austin/Contr/IBM
To: Casey Ching <[email protected]>
Cc: Alex Behm <[email protected]>,
[email protected], Nishidha
Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan
Jagadale/Austin/Contr/IBM@IBMUS, David Clissold/Austin/IBM@IBMUS,
Valencia Serrao/Austin/Contr/IBM@IBMUS
Date: 05/05/2016 02:21 PM
Subject: Re: Fw: Issues with generating testdata for Impala
Thanks, Casey!
I will let you know the test status.
Casey Ching ---05/05/2016 01:09:11 PM---On May 4, 2016 at 11:08:07
PM, Valencia Serrao ([email protected]) wrote: Hi Alex,
From: Casey Ching <[email protected]>
To: Alex Behm <[email protected]>, Valencia
Serrao/Austin/Contr/IBM@IBMUS, [email protected]
Cc: Sudarshan Jagadale/Austin/Contr/IBM@IBMUS, Nishidha
Panpaliya/Austin/Contr/IBM@IBMUS, [email protected]
Date: 05/05/2016 01:09 PM
Subject: Re: Fw: Issues with generating testdata for Impala
On May 4, 2016 at 11:08:07 PM, Valencia Serrao ([email protected])
wrote:
Hi Alex,
I've placed the individual testdata tars at the
IMPALA_HOME/testdata/impala-data. Steps 1...10 i've already
executed. Some queries about step no:11 and step no:12, that
i want to clarify:
1) . bin/impala-config.sh
2) mkdir -p $IMPALA_HOME/testdata/impala-data
3) pushd $IMPALA_HOME/testdata/impala-data
4) cat /tmp/tpch.tar.gz{0..6} > tpch.tar.gz
5) tar -xzf tpch.tar.gz
6) rm tpch.tar.gz
7) cat /tmp/tpcds.tar.gz{0..3} > tpcds.tar.gz
8) tar -xzf tpcds.tar.gz
9) rm tpcds.tar.gz
10) popd
11) ./buildall.sh -notests -noclean -format
-----Here I've removed the -testdata option.
The reason to do this is to clear the previously generated
partial schemas.
I think the -format option is supposed to clear out any old state.
The -testdata flag is probably needed to generate and load the test
data.
12) sudo rm -rf $IMPALA_HOME/testdata/impala-data ---- Is
this step required? Why?
That is only for docker. It helps to reduct the image size. You
shouldn’t need to do that or any of the other rm commands.
Could you kindly confirm on these steps ? If any corrections,
please let me know.
Regards,
Valencia
Valencia Serrao---05/04/2016 04:18:24 PM---Hi Alex/Casey
Thank you for responding and for sharing the testdata. I'm
working on using the testda
From: Valencia Serrao/Austin/Contr/IBM
To: Alex Behm <[email protected]>
Cc: Casey Ching <[email protected]>,
[email protected], Nishidha
Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan
Jagadale/Austin/Contr/IBM@IBMUS, David
Clissold/Austin/IBM@IBMUS
Date: 05/04/2016 04:18 PM
Subject: Re: Fw: Issues with generating testdata for Impala
Hi Alex/Casey
Thank you for responding and for sharing the testdata. I'm
working on using the testdata to run the fe tests.
Meanwhile, I've posted the logs onto "Impala Dev" google
group. Here's the link:
https://groups.google.com/a/cloudera.org/forum/#!topic/impala-dev/zy05cHNrACk
Regards,
Valencia
Alex Behm ---05/04/2016 12:52:44 PM---Ahh, thanks Casey. Did
not know about that. Valencia, Impala's data loading expects
the files to be
From: Alex Behm <[email protected]>
To: Casey Ching <[email protected]>
Cc: [email protected], Sudarshan
Jagadale/Austin/Contr/IBM@IBMUS, Nishidha
Panpaliya/Austin/Contr/IBM@IBMUS, Valencia
Serrao/Austin/Contr/IBM@IBMUS
Date: 05/04/2016 12:52 PM
Subject: Re: Fw: Issues with generating testdata for Impala
Ahh, thanks Casey. Did not know about that.
Valencia, Impala's data loading expects the files to be
placed in IMPALA_HOME/testdata/impala-data
On Tue, May 3, 2016 at 11:21 PM, Casey Ching <
[email protected]> wrote:
Comment inline below
On May 3, 2016 at 11:18:06 PM, Alex Behm (
[email protected]) wrote:
Hi Valencia,
I'm sorry you are having so much trouble with
our setup. Let's see what we
can do.
There was an infra issue with receiving the
logs you sent me. The
email/attachment got rejected on our side.
Maybe you can upload the logs
somewhere so I can grab them?
See more responses inline below.
On Sat, Apr 30, 2016 at 5:01 AM, Valencia
Serrao <[email protected]> wrote:
> Hi Alex,
>
> I was going more deeper through the logs. I
have some findings and queries:
>
> 1. At the "Invalidating Metadata" step (as
mentioned in below mail), i
> noticed that, it is trying to use kerberos.
Perhaps, this is preventing the
> testdata generation from proceeding, as we
are not using Kerberos.
> I need to know how this can be done without
involving Kerberos support ?
>
Kerberos is certainly not needed to build and
run tests.
>
> 2. I had executed the fe tests despite the
incomplete testdata generation,
> the tests started and surely have failed.
Many of these (null pointer
> exception in AuthorzationTests) have a
common cause: "tpch database does
> not exist."
> e.g. as shown
in
.Impala/cluster_logs/query_tests/test-run-workload.log.
>
> Does the "tpch" database gets created after
the current blocker step
> "Invalidating Metadata" ?
>
Yes, the TPCH database is created and loaded
as part of that first phase.
However, the data files are not yet publicly
accessible. Let me work on
that from my side, and get back to you soon.
One way or the other we'll be
able to provide you with the data.
The data is at
https://github.com/cloudera/Impala-docker-hub/tree/master/prereqs/container_root/tmp
. The files are split into 50 MB pieces for git. You can
put them back together as is done in
https://github.com/cloudera/Impala-docker-hub/blob/master/complete/Dockerfile
>
> 3. In the fe test console output log,
another error shown:
> ============================= test session
starts
> ==============================
> platform linux2 -- Python 2.7.5 --
py-1.4.30 -- pytest-2.7.2
> rootdir: /work/, inifile:
> plugins: random, xdist
> ERROR: file not found:/work/I
>
mpala/../Impala-auxiliary-tests/tests/aux_custom_cluster_tests/
>
> These are not present/created on my vm. May
i know when these get created ?
>
> 4. Could you also share the total number of
fe tests ?
>
I'll privately send you the console output
from a successful FE run.
Hopefully that can help.
Cheers,
Alex
>
>
> Looking forward to your reply.
>
> Regards,
> Valencia
>
>
> [image: Inactive hide details for Valencia
Serrao---04/30/2016 09:05:54
> AM---Hi Alex, I've been able to make some
progress on testdata]Valencia
> Serrao---04/30/2016 09:05:54 AM---Hi Alex,
I've been able to make some
> progress on testdata generation, however, i
still face the foll
>
> From: Valencia Serrao/Austin/Contr/IBM
> To: [email protected], Alex
Behm <[email protected]>
> Cc: Sudarshan
Jagadale/Austin/Contr/IBM@IBMUS, Nishidha
> Panpaliya/Austin/Contr/IBM@IBMUS, Valencia
Serrao/Austin/Contr/IBM@IBMUS
> Date: 04/30/2016 09:05 AM
> Subject: Fw: Issues with generating
testdata for Impala
> ------------------------------
>
>
>
> Hi Alex,
>
> I've been able to make some progress on
testdata generation, however, i
> still face the following issues:
>
>
>
*******************************************************************************************************************************************************************
> Invalidating Metadata
>
>
(load-functional-query-exhaustive-impala-load-generated-parquet-none-none.sql):
> INSERT OVERWRITE TABLE
functional_parquet.alltypes partition (year,
month)
> SELECT id, bool_col, tinyint_col,
smallint_col, int_col, bigint_col,
> float_col, double_col, date_string_col,
string_col, timestamp_col, year,
> month
> FROM functional.alltypes
>
> Data Loading from Impala failed with error:
ImpalaBeeswaxException:
> INNER EXCEPTION: <class 'socket.error'>
> MESSAGE: [Errno 104] Connection reset by
peer
> Error
in
/root/nishidha/Impala/testdata/bin/create-load-data.sh
at line
> 41: while [ -n "$*" ]
> Error in /root/nishidha/Impala/buildall.sh
at line 368:
> $
{IMPALA_HOME}/testdata/bin/create-load-data.sh
${CREATE_LOAD_DATA_ARGS}
> <<< Y
>
>
*************************************************************************************************************************************************************************
>
> i continued with fe tests as is. Here is
the complete output log.
> [attachment "fe_test_output.zip" deleted by
Valencia
> Serrao/Austin/Contr/IBM]
>
> Cluster logs: [attachment "cluster_logs.7z"
deleted by Valencia
> Serrao/Austin/Contr/IBM]
>
> Kindly guide me on the same.
>
> Regards,
> Valencia
> ----- Forwarded by Valencia
Serrao/Austin/Contr/IBM on 04/29/2016 10:57
AM
> -----
>
> From: Sudarshan Jagadale/Austin/Contr/IBM
> To: Valencia Serrao/Austin/Contr/IBM@IBMUS
> Date: 04/29/2016 10:49 AM
> Subject: Fw: Issues with generating
testdata for Impala
> ------------------------------
>
>
> FYI
> Thanks and Regards
> Sudarshan Jagadale
> Power Open Source Solutions
> ----- Forwarded by Sudarshan
Jagadale/Austin/Contr/IBM on 04/29/2016 10:48
> AM -----
>
> From: Alex Behm <[email protected]>
> To: [email protected]
> Cc: Sudarshan
Jagadale/Austin/Contr/IBM@IBMUS, Nishidha
> Panpaliya/Austin/Contr/IBM@IBMUS
> Date: 04/28/2016 09:34 PM
> Subject: Re: Issues with generating
testdata for Impala
> ------------------------------
>
>
>
> Hi Valencia,
>
> sorry I did not get the attachment. Would
you be able to tar.gz and attach
> the whole cluster_logs directory?
>
> Alex
>
> On Thu, Apr 28, 2016 at 6:23 AM, Valencia
Serrao <*[email protected]*
> <[email protected]>> wrote:
>
> Hi Alex,
>
> I tried building impala again with the
following:
> HDFS CDH 5.7.0 (
> *
http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_57.html#topic_3*
> <
http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_57.html#topic_3
>
> )
> HBASE CDH 5.7.0 SNAPSHOT (
> *
http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.7.0.tar.gz*
> <
http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.7.0.tar.gz
> )
> - this required to patch in a fix (
> *
https://issues.apache.org/jira/secure/attachment/12792536/HBASE-15322-branch-1.2.patch*
> <
https://issues.apache.org/jira/secure/attachment/12792536/HBASE-15322-branch-1.2.patch
>
> )
> HIVE CDH 5.8.0 SNAPSHOT
>
> With the above combination, i'm able to
move past the exception and
> also have the RegionServer service up and
running. However, it now gives
> error as below:
>
>
>
********************************************************************************************************************
>
(load-functional-query-exhaustive-impala-generated-text-none-none.sql):
> CREATE EXTERNAL TABLE IF NOT EXISTS
functional.decimal_tbl (
> d1 DECIMAL,
> d2 DECIMAL(10, 0),
> d3 DECIMAL(20, 10),
> d4 DECIMAL(38, 38),
> d5 DECIMAL(10, 5))
> PARTITIONED BY (d6 DECIMAL(9, 0))
> ROW FORMAT delimited fields terminated by
','
> STORED AS TEXTFILE
> LOCATION '/test-warehouse/decimal_tbl'
>
>
(load-functional-query-exhaustive-impala-generated-text-none-none.sql):
> USE functional
>
>
(load-functional-query-exhaustive-impala-generated-text-none-none.sql):
> ALTER TABLE decimal_tbl ADD IF NOT EXISTS
PARTITION(d6=1)
>
> Data Loading from Impala failed with error:
ImpalaBeeswaxException:
> INNER EXCEPTION: <class
>
'impala._thrift_gen.beeswax.ttypes.BeeswaxException'>
> MESSAGE:
> Error: null
>
>
******************************************************************************************************************
>
> Here is the complete log for the same.
*(See attached file:
> data-load-functional-exhaustive.log)*
>
> It would great if you could guide me on
this issue, so i could proceed
> with the fe tests.
>
> Still awaiting link to the source code of
HDFS CDH 5.8.0
>
> Regards,
> Valencia
>
>
>
>
Split and assign HBase regions (logging to split-hbase.log)... OK
Starting Impala cluster (logging to start-impala-cluster.log)... OK
Run test run-workload (logging to test-run-workload.log)... OK
============================= test session starts ==============================
platform linux2 -- Python 2.7.5 -- py-1.4.30 -- pytest-2.7.2
rootdir: , inifile:
plugins: random, xdist
collected 41 items
custom_cluster/test_admission_controller.py .F...
custom_cluster/test_alloc_fail.py ..
custom_cluster/test_delegation.py ...
custom_cluster/test_exchange_delays.py ..
custom_cluster/test_hdfs_fd_caching.py .
custom_cluster/test_hive_parquet_timestamp_conversion.py FF
custom_cluster/test_insert_behaviour.py ..
custom_cluster/test_legacy_joins_aggs.py .
custom_cluster/test_parquet_max_page_header.py .
custom_cluster/test_permanent_udfs.py ...
custom_cluster/test_query_expiration.py ...
custom_cluster/test_redaction.py ....
custom_cluster/test_scratch_disk.py ..F.
custom_cluster/test_session_expiration.py .
custom_cluster/test_spilling.py ....
authorization/test_authorization.py ..
authorization/test_grant_revoke.py F
=================================== FAILURES ===================================
________________ TestAdmissionController.test_set_request_pool _________________
self = <test_admission_controller.TestAdmissionController object at
0x10035b71590>
def add_session(self):
open_session_req = TCLIService.TOpenSessionReq()
open_session_req.username = getuser()
open_session_req.configuration = dict()
if conf_overlay is not None:
open_session_req.configuration = conf_overlay
open_session_req.client_protocol = protocol_version
resp = self.hs2_client.OpenSession(open_session_req)
HS2TestSuite.check_response(resp)
self.session_handle = resp.sessionHandle
assert protocol_version <= resp.serverProtocolVersion
try:
> fn(self)
hs2/hs2_test_suite.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
custom_cluster/test_admission_controller.py:189: in test_set_request_pool
self.__check_pool_rejected(client, 'user.queueC', expected_error)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <test_admission_controller.TestAdmissionController object at
0x10035b71590>
client = <tests.common.impala_connection.BeeswaxConnection object at
0x10035b6db90>
pool = 'user.queueC'
expected_error_re = "Request from user '\\w+' with requested pool 'user.queueC'
denied access to assigned pool 'user.queueC'"
def __check_pool_rejected(self, client, pool, expected_error_re):
try:
client.set_configuration({'request_pool': pool})
client.execute("select 1")
assert False, "Query should return error"
except ImpalaBeeswaxException as e:
> assert re.search(expected_error_re, str(e))
E assert None
E + where None = <function search at 0x100344009b0>("Request from user
'\\w+' with requested pool 'user.queueC' denied access to assigned pool
'user.queueC'", "ImpalaBeeswaxException:\n INNER EXCEPTION: <class
'impala._thrift_gen.beeswax.ttypes.BeeswaxException'>\n MESSAGE: Re...M_LIMIT
times the number of nodes executing the query. See the Admission Control
documentation for more information.\n")
E + where <function search at 0x100344009b0> = re.search
E + and "ImpalaBeeswaxException:\n INNER EXCEPTION: <class
'impala._thrift_gen.beeswax.ttypes.BeeswaxException'>\n MESSAGE: Re...M_LIMIT
times the number of nodes executing the query. See the Admission Control
documentation for more information.\n" = str(ImpalaBeeswaxException())
custom_cluster/test_admission_controller.py:119: AssertionError
---------------------------- Captured stdout setup -----------------------------
Starting State Store logging to
/ImpalaPPC/cluster_logs/custom_cluster//statestored.INFO
Starting Catalog Service logging to
/ImpalaPPC/cluster_logs/custom_cluster//catalogd.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad_node1.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad_node2.INFO
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Impala Cluster Running with 3 nodes.
---------------------------- Captured stderr setup -----------------------------
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: Waiting for num_known_live_backends=3. Current value: 0
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: Waiting for num_known_live_backends=3. Current value: 2
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: Waiting for num_known_live_backends=3. Current value: 2
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25002
MainThread: num_known_live_backends has reached value: 3
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting metric: statestore.live-backends from testvm:25010
MainThread: Metric 'statestore.live-backends' has reach desired value: 4
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25002
MainThread: num_known_live_backends has reached value: 3
-- connecting to: localhost:21000
-- executing against localhost:21000
use default;
SET sync_ddl=1;
-- executing against localhost:21000
drop database if exists `hs2_db` cascade;
----------------------------- Captured stderr call -----------------------------
-- connecting to: testvm:21000
SET request_pool=;
-- executing against testvm:21000
select 1;
SET request_pool=not_a_pool_name;
-- executing against testvm:21000
select 1;
SET request_pool=user.queueC;
-- executing against testvm:21000
select 1;
-- closing connection to: testvm:21000
TestHiveParquetTimestampConversion.test_conversion[exec_option:
{'disable_codegen': False, 'abort_on_error': 1,
'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} |
table_format: parquet/none]
self =
<test_hive_parquet_timestamp_conversion.TestHiveParquetTimestampConversion
object at 0x10035c60990>
vector = <tests.common.test_vector.TestVector object at 0x10035a4a790>
@pytest.mark.execute_serially
@CustomClusterTestSuite.with_args("-convert_legacy_hive_parquet_utc_timestamps=true")
def test_conversion(self, vector):
tz_name = time.tzname[time.localtime().tm_isdst]
> self.check_sanity(tz_name not in ("UTC", "GMT"))
custom_cluster/test_hive_parquet_timestamp_conversion.py:62:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
custom_cluster/test_hive_parquet_timestamp_conversion.py:40: in check_sanity
FROM functional_parquet.alltypesagg_hive_13_1""")\
common/impala_test_suite.py:359: in wrapper
return function(*args, **kwargs)
common/impala_test_suite.py:366: in execute_query_expect_success
result = cls.__execute_query(impalad_client, query, query_options)
common/impala_test_suite.py:451: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:161: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:163: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:329: in __execute_query
self.wait_for_completion(handle)
beeswax/impala_beeswax.py:341: in wait_for_completion
query_state = self.get_state(query_handle)
beeswax/impala_beeswax.py:358: in get_state
return self.__do_rpc(lambda: self.imp_service.get_state(query_handle))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tests.beeswax.impala_beeswax.ImpalaBeeswaxClient object at
0x10035c60fd0>
rpc = <function <lambda> at 0x10034a959b0>
def __do_rpc(self, rpc):
"""Executes the RPC lambda provided with some error checking.
Catches all the relevant exceptions and re throws them wrapped
in a custom exception [ImpalaBeeswaxException].
"""
if not self.connected:
raise ImpalaBeeswaxException("Not connected", None)
try:
return rpc()
except BeeswaxService.BeeswaxException, b:
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
except TTransportException, e:
self.connected = False
raise ImpalaBeeswaxException(self.__build_error_message(e), e)
except TApplicationException, t:
raise ImpalaBeeswaxException(self.__build_error_message(t), t)
except Exception, u:
> raise ImpalaBeeswaxException(self.__build_error_message(u), u)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
E INNER EXCEPTION: <class 'socket.error'>
E MESSAGE: [Errno 104] Connection reset by peer
beeswax/impala_beeswax.py:455: ImpalaBeeswaxException
---------------------------- Captured stdout setup -----------------------------
Starting State Store logging to
/ImpalaPPC/cluster_logs/custom_cluster//statestored.INFO
Starting Catalog Service logging to
/ImpalaPPC/cluster_logs/custom_cluster//catalogd.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad_node1.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad_node2.INFO
Waiting for Catalog... Status: 1 DBs / 0 tables (ready=False)
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Impala Cluster Running with 3 nodes.
---------------------------- Captured stderr setup -----------------------------
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: Waiting for num_known_live_backends=3. Current value: 0
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25002
MainThread: num_known_live_backends has reached value: 3
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting metric: statestore.live-backends from testvm:25010
MainThread: Metric 'statestore.live-backends' has reach desired value: 4
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25002
MainThread: num_known_live_backends has reached value: 3
-- connecting to: localhost:21000
----------------------------- Captured stderr call -----------------------------
-- executing against localhost:21000
SELECT COUNT(timestamp_col), COUNT(DISTINCT timestamp_col),
MIN(timestamp_col), MAX(timestamp_col)
FROM functional_parquet.alltypesagg_hive_13_1;
TestHiveParquetTimestampConversion.test_no_conversion[exec_option:
{'disable_codegen': False, 'abort_on_error': 1,
'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} |
table_format: parquet/none]
self =
<test_hive_parquet_timestamp_conversion.TestHiveParquetTimestampConversion
object at 0x10035db9690>
vector = <tests.common.test_vector.TestVector object at 0x10035a4a9d0>
@pytest.mark.execute_serially
@CustomClusterTestSuite.with_args("-convert_legacy_hive_parquet_utc_timestamps=false")
def test_no_conversion(self, vector):
> self.check_sanity(False)
custom_cluster/test_hive_parquet_timestamp_conversion.py:82:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
custom_cluster/test_hive_parquet_timestamp_conversion.py:40: in check_sanity
FROM functional_parquet.alltypesagg_hive_13_1""")\
common/impala_test_suite.py:359: in wrapper
return function(*args, **kwargs)
common/impala_test_suite.py:366: in execute_query_expect_success
result = cls.__execute_query(impalad_client, query, query_options)
common/impala_test_suite.py:451: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:161: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:163: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:329: in __execute_query
self.wait_for_completion(handle)
beeswax/impala_beeswax.py:341: in wait_for_completion
query_state = self.get_state(query_handle)
beeswax/impala_beeswax.py:358: in get_state
return self.__do_rpc(lambda: self.imp_service.get_state(query_handle))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tests.beeswax.impala_beeswax.ImpalaBeeswaxClient object at
0x10035c60490>
rpc = <function <lambda> at 0x10035c6a7d0>
def __do_rpc(self, rpc):
"""Executes the RPC lambda provided with some error checking.
Catches all the relevant exceptions and re throws them wrapped
in a custom exception [ImpalaBeeswaxException].
"""
if not self.connected:
raise ImpalaBeeswaxException("Not connected", None)
try:
return rpc()
except BeeswaxService.BeeswaxException, b:
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
except TTransportException, e:
self.connected = False
raise ImpalaBeeswaxException(self.__build_error_message(e), e)
except TApplicationException, t:
raise ImpalaBeeswaxException(self.__build_error_message(t), t)
except Exception, u:
> raise ImpalaBeeswaxException(self.__build_error_message(u), u)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
E INNER EXCEPTION: <class 'socket.error'>
E MESSAGE: [Errno 104] Connection reset by peer
beeswax/impala_beeswax.py:455: ImpalaBeeswaxException
---------------------------- Captured stdout setup -----------------------------
Starting State Store logging to
/ImpalaPPC/cluster_logs/custom_cluster//statestored.INFO
Starting Catalog Service logging to
/ImpalaPPC/cluster_logs/custom_cluster//catalogd.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad_node1.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad_node2.INFO
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Impala Cluster Running with 3 nodes.
---------------------------- Captured stderr setup -----------------------------
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: Waiting for num_known_live_backends=3. Current value: 2
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: Waiting for num_known_live_backends=3. Current value: 2
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: Waiting for num_known_live_backends=3. Current value: 2
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25002
MainThread: num_known_live_backends has reached value: 3
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting metric: statestore.live-backends from testvm:25010
MainThread: Metric 'statestore.live-backends' has reach desired value: 4
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25002
MainThread: num_known_live_backends has reached value: 3
-- connecting to: localhost:21000
----------------------------- Captured stderr call -----------------------------
-- executing against localhost:21000
SELECT COUNT(timestamp_col), COUNT(DISTINCT timestamp_col),
MIN(timestamp_col), MAX(timestamp_col)
FROM functional_parquet.alltypesagg_hive_13_1;
TestScratchDir.test_non_writable_dirs[exec_option: {'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0,
'num_nodes': 0} | table_format: text/none]
self = <test_scratch_disk.TestScratchDir object at 0x10035c8ba50>
vector = <tests.common.test_vector.TestVector object at 0x10035b41ed0>
@pytest.mark.execute_serially
@CustomClusterTestSuite.with_args('-scratch_dirs=%s' %
get_dirs(NON_WRITABLE_DIRS))
def test_non_writable_dirs(self, vector):
""" Test we can execute a query with only bad non-writable scratch """
self.assert_impalad_log_contains("ERROR", "Running without spill to disk:
could "
+ "not use any scratch directories in list:.*. See previous "
> + "warnings for information on causes.")
custom_cluster/test_scratch_disk.py:117:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <test_scratch_disk.TestScratchDir object at 0x10035c8ba50>
level = 'ERROR'
line_regex = 'Running without spill to disk: could not use any scratch
directories in list:.*. See previous warnings for information on causes.'
expected_count = 1
def assert_impalad_log_contains(self, level, line_regex, expected_count=1):
"""
Assert that impalad log with specified level (e.g. ERROR, WARNING, INFO)
contains expected_count lines with a substring matching the regex.
"""
pattern = re.compile(line_regex)
found = 0
log_file_path = os.path.join(self.impala_log_dir, "impalad." + level)
with open(log_file_path) as log_file:
for line in log_file:
if pattern.search(line):
found += 1
assert found == expected_count, ("Expected %d lines in file %s matching
regex '%s'"\
> + ", but found %d lines") % (expected_count, log_file_path,
> line_regex, found)
E AssertionError: Expected 1 lines in file
/ImpalaPPC/cluster_logs/custom_cluster/impalad.ERROR matching regex 'Running
without spill to disk: could not use any scratch directories in list:.*. See
previous warnings for information on causes.', but found 0 lines
common/custom_cluster_test_suite.py:130: AssertionError
---------------------------- Captured stdout setup -----------------------------
Starting State Store logging to
/ImpalaPPC/cluster_logs/custom_cluster//statestored.INFO
Starting Catalog Service logging to
/ImpalaPPC/cluster_logs/custom_cluster//catalogd.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad_node1.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad_node2.INFO
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Impala Cluster Running with 3 nodes.
---------------------------- Captured stderr setup -----------------------------
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: Waiting for num_known_live_backends=3. Current value: 0
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25002
MainThread: Waiting for num_known_live_backends=3. Current value: 2
MainThread: Getting num_known_live_backends from testvm:25002
MainThread: num_known_live_backends has reached value: 3
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting metric: statestore.live-backends from testvm:25010
MainThread: Metric 'statestore.live-backends' has reach desired value: 4
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25002
MainThread: num_known_live_backends has reached value: 3
-- connecting to: localhost:21000
TestGrantRevoke.test_grant_revoke[exec_option: {'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0,
'num_nodes': 0} | table_format: text/none]
self = <test_grant_revoke.TestGrantRevoke object at 0x10035db2810>
vector = <tests.common.test_vector.TestVector object at 0x10035b6d210>
@pytest.mark.execute_serially
@CustomClusterTestSuite.with_args(
impalad_args="--server_name=server1",
catalogd_args="--sentry_config=" + SENTRY_CONFIG_FILE)
def test_grant_revoke(self, vector):
if IS_S3:
self.run_test_case('QueryTest/grant_revoke_no_insert', vector,
use_db="default")
else:
> self.run_test_case('QueryTest/grant_revoke', vector, use_db="default")
authorization/test_grant_revoke.py:81:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
common/impala_test_suite.py:287: in run_test_case
pytest.config.option.update_results)
common/test_result_verifier.py:357: in verify_raw_results
VERIFIER_MAP[verifier](expected, actual)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
expected_results = <tests.common.test_result_verifier.QueryTestResult object at
0x10035dd9d90>
actual_results = <tests.common.test_result_verifier.QueryTestResult object at
0x10035dd9f10>
def verify_query_result_is_subset(expected_results, actual_results):
assert_args_not_none(expected_results, actual_results)
expected_set = set(map(str, expected_results.rows))
actual_set = set(map(str, actual_results.rows))
> assert expected_set <= actual_set
E assert Items in expected results not found in actual results:
common/test_result_verifier.py:199: AssertionError
---------------------------- Captured stdout setup -----------------------------
Starting State Store logging to
/ImpalaPPC/cluster_logs/custom_cluster//statestored.INFO
Starting Catalog Service logging to
/ImpalaPPC/cluster_logs/custom_cluster//catalogd.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad_node1.INFO
Starting Impala Daemon logging to
/ImpalaPPC/cluster_logs/custom_cluster//impalad_node2.INFO
Waiting for Catalog... Status: 1 DBs / 0 tables (ready=False)
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Waiting for Catalog... Status: 39 DBs / 994 tables (ready=True)
Impala Cluster Running with 3 nodes.
---------------------------- Captured stderr setup -----------------------------
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: Waiting for num_known_live_backends=3. Current value: 0
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25002
MainThread: num_known_live_backends has reached value: 3
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting metric: statestore.live-backends from testvm:25010
MainThread: Metric 'statestore.live-backends' has reach desired value: 4
MainThread: Getting num_known_live_backends from testvm:25000
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25001
MainThread: num_known_live_backends has reached value: 3
MainThread: Getting num_known_live_backends from testvm:25002
MainThread: num_known_live_backends has reached value: 3
-- connecting to: localhost:21000
-- executing against localhost:21000
show roles;
-- executing against localhost:21000
show role grant group `user`;
-- executing against localhost:21000
drop role all_functional_seq_snap;
-- executing against localhost:21000
drop role all_newdb;
-- executing against localhost:21000
drop role all_tpch;
-- executing against localhost:21000
drop role insert_functional_alltypes;
-- executing against localhost:21000
drop role insert_parquet;
-- executing against localhost:21000
drop role select_column_level_functional;
-- executing against localhost:21000
drop role select_functional_alltypesagg;
-- executing against localhost:21000
drop role select_functional_complex_view;
-- executing against localhost:21000
drop role select_functional_view_view;
-- executing against localhost:21000
drop role select_tpcds;
-- executing against localhost:21000
drop role testrolename;
-- executing against localhost:21000
create role grant_revoke_test_admin;
-- executing against localhost:21000
grant all on server to grant_revoke_test_admin;
-- executing against localhost:21000
grant role grant_revoke_test_admin to group user;
-- executing against localhost:21000
use default;
SET sync_ddl=0;
-- executing against localhost:21000
drop database if exists `grant_rev_db` cascade;
-- executing against localhost:21000
drop role grant_revoke_test_admin;
----------------------------- Captured stderr call -----------------------------
-- executing against localhost:21000
use default;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET batch_size=0;
SET num_nodes=0;
-- executing against localhost:21000
create role grant_revoke_test_ALL_SERVER;
-- executing against localhost:21000
create role grant_revoke_test_ALL_TEST_DB;
-- executing against localhost:21000
create role grant_revoke_test_SELECT_INSERT_TEST_TBL;
-- executing against localhost:21000
create role grant_revoke_test_ALL_URI;
-- executing against localhost:21000
show roles;
-- executing against localhost:21000
create database grant_rev_db location '/test-warehouse/grant_rev_db.db';
-- executing against localhost:21000
grant all on server to grant_revoke_test_ALL_SERVER;
-- executing against localhost:21000
grant role grant_revoke_test_ALL_SERVER to group user;
-- executing against localhost:21000
show current roles;
-- connecting to: localhost:21000
-- executing against localhost:21000
show current roles;
-- executing against localhost:21000
show grant role grant_revoke_test_ALL_SERVER;
-- executing against localhost:21000
show grant role grant_revoke_test_ALL_SERVER on server;
-- connecting to: localhost:21000
-- executing against localhost:21000
show roles;
-- connecting to: localhost:21000
-- executing against localhost:21000
show role grant group user;
-- connecting to: localhost:21000
-- executing against localhost:21000
show role grant group user;
MainThread: Items in expected results not found in actual results:
generated xml file:
/ImpalaPPC/tests/custom_cluster/results/TEST-impala-custom-cluster.xml
============== 5 failed, 36 passed, 3 warnings in 2574.74 seconds ==============
Error in /ImpalaPPC/tests/run-custom-cluster-tests.sh at line 41:
impala-py.test custom_cluster/ authorization/ ${AUX_CUSTOM_DIR} \
