Hi Tim, To trace the exploration strategy var at each step leading to the "test_spilling" test, I put a few print statements for it. I observed that, once the process flow reaches the impala_test_suite.py the strategy is changed to 'core'. The logs printed as follows: (logs marked pink)
Waiting for HiveServer2 at localhost:11050... Could not connect to localhost:11050 HiveServer2 service is up at localhost:11050 --> Starting the Sentry Policy Server in buildall.sh before run-all-tests.sh:::::::Exploration strategy= exhaustive 1st in run-all-tests.sh :::::::Exploration strategy= exhaustive Split and assign HBase regions (logging to split-hbase.log)... OK Starting Impala cluster (logging to start-impala-cluster.log)... OK Run test run-workload (logging to test-run-workload.log)... OK Starting CC tests:::::::Exploration strategy= exhaustive ============================= test session starts ============================== platform linux2 -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2 rootdir: /home/test/ProjectImpala, inifile: plugins: xdist, random default_strategy in impala_test_suite.py::::EXPLORATION STRATEGY= core default_strategy in impala_test_suite.py::::EXPLORATION STRATEGY= core in conftest.py:::::Exploration Strategy = core in test_vector:::::Exploration Strategy = core default_strategy in impala_test_suite.py::::EXPLORATION STRATEGY= core default_strategy in impala_test_suite.py::::EXPLORATION STRATEGY= core in conftest.py:::::Exploration Strategy = core in test_vector.py:::::Exploration Strategy = core collected 4 items custom_cluster/test_spilling.py sss. generated xml file: /home/test/ProjectImpala/ImpalaPPC/tests/custom_cluster/results/TEST-impala-custom-cluster.xml ============== 1 passed, 3 skipped, 1 warnings in 110.42 seconds =============== Could you please guide me on this issue ? Regards, Valencia From: Valencia Serrao/Austin/Contr/IBM To: Nishidha Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan Jagadale/Austin/Contr/IBM@IBMUS, Manish Patil/Austin/Contr/IBM@IBMUS Date: 08/02/2016 10:49 AM Subject: Fw: Issues with tests in Release-mode Impala build ----- Forwarded by Valencia Serrao/Austin/Contr/IBM on 08/02/2016 10:48 AM ----- From: Valencia Serrao/Austin/Contr/IBM To: Tim Armstrong <tarmstr...@cloudera.com> Date: 08/02/2016 09:05 AM Subject: Re: Issues with tests in Release-mode Impala build Hi Tim, I am executing the tests in exhaustive mode, but still I see that the "test_spilling" test still fails with skip message "runs only in exhaustive." Following are the various ways I tried to run the tests: 1. ${IMPALA_HOME}/buildall.sh -noclean -testexhaustive 2. Explicitly set EXPLORATION_STRATEGY in run-all_tests.sh and buildall.sh to exhaustive. I think it is getting reset somewhere to some other strategy. Could you please help me to correctly set the env to run the Custom cluster in the exhaustive exploration strategy ? Regards, Valencia From: Valencia Serrao/Austin/Contr/IBM To: Tim Armstrong <tarmstr...@cloudera.com> Cc: dev@impala.incubator.apache.org, Manish Patil/Austin/Contr/IBM@IBMUS, Nishidha Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan Jagadale/Austin/Contr/IBM@IBMUS, Valencia Serrao/Austin/Contr/IBM@IBMUS Date: 07/25/2016 05:56 PM Subject: Re: Issues with tests in Release-mode Impala build Hi Tim, Thanks for the detailed response. Also, the BE "benchmark-test" issue is resolved. It now passes together with the complete BE suite in Release mode. Regards, Valencia From: Tim Armstrong <tarmstr...@cloudera.com> To: Valencia Serrao/Austin/Contr/IBM@IBMUS Cc: dev@impala.incubator.apache.org, Manish Patil/Austin/Contr/IBM@IBMUS, Nishidha Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan Jagadale/Austin/Contr/IBM@IBMUS Date: 07/23/2016 12:15 AM Subject: Re: Issues with tests in Release-mode Impala build 2a. Exhaustive is a superset of core. We run the core tests pre-commit on CentOS 6 + HDFS and the full exhaustive tests post-commit on a wider range of configurations. We don't release Impala unless all exhaustive tests passed on all configurations we test (if there's a valid reason why something doesn't work on a given platform we skip the test). 2b. Exhaustive is a superset of core, so if exhaustive passes then core should do. The exhaustive build takes much longer than core so it makes sense to run it less frequently (e.g. we run it nightly for some configurations and weekly for others). 2c. Confusingly, the core/exhaustive data load doesn't map to core/exhaustive tests. We actually use the same data load for all test configurations. See testdata/bin/create-load-data.sh for how the core/exhaustive data load is invoked. E.g. we load the functional data with exhaustive (i.e. all supported file formats) and the larger tpc-h/tpc-ds data sets for only a subset of file forms. On Wed, Jul 20, 2016 at 9:39 PM, Valencia Serrao <vser...@us.ibm.com> wrote: Hi Tim, Thank you for the insight on the issues. 1. BE test -issue: benchmark-test hangs As you suggested, I increased the "batch_size" value to upto 125000000, however, the sw.ElapsedTime() does not increase inside the while and again gets caught up in an infinite loop. The optimization level seems to cause this behavior. I am still working in this. 2. Custom cluster tests: skipping some tests in test_spilling I found in the logs that the "test_spilling" test was skipped as the exploration strategy was set to "core" on our Impala setup. Some question here, a. From a Impala release perspective how significant are these strategies (core, exhaustive, etc.) ? b. Do we have to test with all combinations (core|release mode build and exhaustive|release mode build). c. Does the exploration strategy selection also affect the test data loaded ? (data loaded is different in each exploration strategy ? ) Please let me know your comments. Regards, Valencia Inactive hide details for Tim Armstrong ---07/19/2016 09:11:48 PM---With 2, it's a little strange that test_spilling is being sTim Armstrong ---07/19/2016 09:11:48 PM---With 2, it's a little strange that test_spilling is being skipped - I think that one should be run. From: Tim Armstrong <tarmstr...@cloudera.com> To: Valencia Serrao/Austin/Contr/IBM@IBMUS Cc: dev@impala.incubator.apache.org, Manish Patil/Austin/Contr/IBM@IBMUS, Nishidha Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan Jagadale/Austin/Contr/IBM@IBMUS Date: 07/19/2016 09:11 PM Subject: Re: Issues with tests in Release-mode Impala build With 2, it's a little strange that test_spilling is being skipped - I think that one should be run. On Tue, Jul 19, 2016 at 8:39 AM, Tim Armstrong <tarmstr...@cloudera.com> wrote: It looks like the benchmark-test issue is something to do with the granularity of the clock. It can get stuck in an infinite loop if the function call below always takes less than the smallest measurable unit of time (i.e. Start() and Stop() are called in the same time quantum). while (sw.ElapsedTime() < target_cycles) { sw.Start(); function(batch_size, args); sw.Stop(); iters += batch_size; } We use Intel's rdtsc instruction for a timer here, so I guess whatever PPC alternative you used may work a little differently. This is probably ok, but it's possible that it could affect timers elsewhere in Impala. One solution would be to increase the default batch size. On Tue, Jul 19, 2016 at 5:29 AM, Valencia Serrao < vser...@us.ibm.com> wrote: Hi Tim, Following are some observations: 1. BE test -issue: benchmark-test hangs Putting trace logs like below in benchmark.cc: while (sw.ElapsedTime() < target_cycles) { LOG(INFO) <<" in while(sw.ElapsedTime() < target_cycles)"; sw.Start(); function(batch_size, args); sw.Stop(); iters += batch_size; LOG(INFO) <<" In while:::::::: sw.ElapsedTime() "<< sw.ElapsedTime (); LOG(INFO) <<" In while:::::::: iters = " << iters ; In Release mode, I observed that the sw.ElapsedTime() remains constant and does not increase, therefore, it is caught up in an infinite loop and the benchmark-test hangs. In Debug mode, sw.ElapsedTime() keeps on increasing and therefore is able to come out of the while loop and benchmark-test doesn't hang in Debug mode. I'm working on this issue, however, if you could give any pointers about it, that would be really great. 2. Custom cluster tests: I have included the code changes in my branch and many of the earlier 36 skipped tests have now executed and they pass, but with the following exception(when compared to the output in thehttps://issues.cloudera.org/browse/IMPALA-3614 ): custom_cluster/test_spilling.py sss. Current CC test stats: 34 passed, 7 skipped, 3 warnings. 3. End-to-End tests: I couldn't dive into the EE tests. I will surely let you know more about them as soon as I'm done with them. Regards, Valencia Inactive hide details for Valencia Serrao---07/19/2016 10:26:31 AM---Hi Tim, Thank you for the information.Valencia Serrao---07/19/2016 10:26:31 AM---Hi Tim, Thank you for the information. From: Valencia Serrao/Austin/Contr/IBM To: Tim Armstrong <tarmstr...@cloudera.com> Cc: dev@impala.incubator.apache.org, Manish Patil/Austin/Contr/IBM@IBMUS, Nishidha Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan Jagadale/Austin/Contr/IBM@IBMUS Date: 07/19/2016 10:26 AM Subject: Re: Issues with tests in Release-mode Impala build Hi Tim, Thank you for the information. I am working on the pointers you have given and also on the fix for Custom cluster (skipped) tests. I will inform you on the findings. Regards, Valencia Inactive hide details for Tim Armstrong ---07/18/2016 09:19:52 PM---Hi Valencia, 1. We run tests in release mode nightly and itTim Armstrong ---07/18/2016 09:19:52 PM---Hi Valencia, 1. We run tests in release mode nightly and it doesn't look like we've seen From: Tim Armstrong <tarmstr...@cloudera.com> To: dev@impala.incubator.apache.org Cc: Valencia Serrao/Austin/Contr/IBM@IBMUS, Nishidha Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan Jagadale/Austin/Contr/IBM@IBMUS, Manish Patil/Austin/Contr/IBM@IBMUS Date: 07/18/2016 09:19 PM Subject: Re: Issues with tests in Release-mode Impala build Hi Valencia, 1. We run tests in release mode nightly and it doesn't look like we've seen this hang. I'd suggest you attach a debugger to the benchmark-test process and see what it's doing. It could either be an actual hang, or an infinite/very long loop. That test is only testing our benchmarking utilities, not Impala itself, but IMO it's always good to understand why something like that is happening in case there's a more general problem. 2. Sounds like https://issues.cloudera.org/browse/IMPALA-3614 . Have you got the fix for that in your branch? 3. Look forward to hearing more. Cheers, Tim On Mon, Jul 18, 2016 at 2:49 AM, Valencia Serrao < vser...@us.ibm.com> wrote: Hi All, I have built Impala in Release mode. I executed the tests, following are some observations: 1. BE test: The test execution hangs at the "benchmark-test". There are no errors shown and it hangs at this test. Earlier, running the BE tests in debug mode this issue did not occur. 2. Custom Cluster test: 5 tests passed and 36 tests skipped. All of the skipped cases give the message: "INSERT not implemented for S3" 3. EE tests: I've also seen some failures here (yet to check the details) As for FE and JDBC tests, everything works fine, release mode test output is same as that of debug mode test output. Is the "benchmark-test" test known to fail in Release mode or am I missing out on any configuration. Also, I want to understand the significance of this test, if in case we could ignore it and move ahead. Regards, Valencia