If you haven't already, I'd suggest rebasing on cdh5-trunk and taking a look at my patch here that fixes those tests for later versions of LLVM (on x86): http://gerrit.cloudera.org/#/c/2486/ . There were a lot of subtle issues so it will save you a lot of time.
- Tim On Wed, Mar 23, 2016 at 9:45 AM, Jim Apple <[email protected]> wrote: > You might try looking for your broken tests in the bug tracker. For > instance, looking for expr-test leads to > https://issues.cloudera.org/browse/IMPALA-2995. > > On Wed, Mar 23, 2016 at 2:33 AM, Nishidha Panpaliya <[email protected]> > wrote: > >> Hi Tim and Jim, >> >> Once again I thank you for your quick help. >> >> I ran run-backend-tests.sh and here is the result of all tests - >> >> 89% tests passed, 8 tests failed out of 71 >> >> Total Test time (real) = 979.08 sec >> >> The following tests FAILED: >> 1 - llvm-codegen-test (SEGFAULT) >> 13 - expr-test (OTHER_FAULT) >> 14 - expr-codegen-test (OTHER_FAULT) >> 19 - data-stream-test (Failed) >> 22 - buffered-block-mgr-test (Failed) >> 32 - tmp-file-mgr-test (Failed) >> 33 - row-batch-serialize-test (SEGFAULT) >> 68 - filesystem-util-test (Failed) >> >> >> PFA full log. >> *(See attached file: LastTest.log)* >> >> I also investigated some of the failures like >> >> - *In filesystem-util-test* : Cause of this failure is user running >> the tests being "root". I could not run these tests with non-root user but >> I tested a sample test application that does exactly same thing as done in >> Createdirectory test of filesystem-util-test and it passed with non-root >> user. Also verified this by linux commands mkdir, chmod, rmdir with root >> and non-root user. >> - *In tmp-file-mgr-test *: Even this failure looks same as it also >> does chmod and the tries allocating space. Since I'm running these tests >> using root user, I would not get failure in accessing the dir/file even >> after removing write permissions if the user is root. >> - *In* *llvm-codegen-test: *I tried debugging this test and found a >> crash in llvm::Type::getVoidTy(...). >> >> I'll keep on investigating other crashes and segmentation faults. But in >> case, you find any of the failures familiar or existing on x86 platforms, >> please let us know. >> >> Another news is we have got approval to share our patches. Soon, I'll be >> uploading a patch with LLVM up-gradation work. >> >> Thanks, >> Nishidha >> >> >> Thanks, >> Nishidha >> [image: Inactive hide details for Tim Armstrong ---03/21/2016 09:10:46 >> PM---We generally run the full test suite on machines with at le]Tim >> Armstrong ---03/21/2016 09:10:46 PM---We generally run the full test suite >> on machines with at least 32GB of memory: it's pretty memory hu >> >> From: Tim Armstrong <[email protected]> >> To: Jim Apple <[email protected]> >> Cc: [email protected], Sudarshan >> Jagadale/Austin/Contr/IBM@IBMUS, Manish Patil/Austin/Contr/IBM@IBMUS, >> David Clissold/Austin/IBM@IBMUS, Nishidha >> Panpaliya/Austin/Contr/IBM@IBMUS >> Date: 03/21/2016 09:10 PM >> Subject: Re: Fw: Debugging Impala code >> ------------------------------ >> >> >> >> We generally run the full test suite on machines with at least 32GB of >> memory: it's pretty memory hungry because you have 3 Impalads running >> side-by-side. I believe we tend to run the full data load on machines with >> even more memory. You can start the test cluster with a single impalad >> before running tests (./bin/start-impala-cluster -s1 && >> ./tests/run-tests.py). Some tests will fail since they assume 3 Impalads >> but most should work ok. >> >> Starting with the backend tests sounds like a good idea - they do >> exercise some of the codegen and other architecture-dependent parts that >> will likely be tricky. >> >> - Tim >> >> On Mon, Mar 21, 2016 at 5:09 AM, Jim Apple <*[email protected]* >> <[email protected]>> wrote: >> >> I think you should be able to run the backend tests without data >> loading: >> >> ./bin/run-backend-tests.sh >> # or >> ctest >> >> As in the frontend tests, you can specify which test you want to run: >> >> ctest --output-on-failure -R expr-test # also shows what broke, if >> anything >> >> To only build the backend test run: >> >> make be-test >> >> On Mon, Mar 21, 2016 at 4:12 AM, Nishidha Panpaliya < >> *[email protected]* <[email protected]>> wrote: >> Thanks Jim and Tim for your replies. Really appreciate your >> co-operation and promptness. >> >> I've a few more queries - >> >> 1. What is the memory requirement of Impala to run all the tests? >> Currently, I see test data creation and loading is consuming almost 7GB of >> RAM. And after this, it gets stopped with bad_alloc exception. I've >> already >> requested to increase RAM of my VM. But just wanted to know if 16GB will >> suffice. >> >> 2. Can we skip load testing at this stage and simply run basic unit >> tests at first? Or is there any setting by means of which we can lower the >> volume of test data being generated/loaded? Once basic tests are working, >> we can focus on load testing. >> >> Also, we wish to have a call with you to discuss all this. We are >> located in India. >> >> Thanks, >> Nishidha >> >> >> Sudarshan Jagadale---03/18/2016 11:04:45 AM---Thanks and Regards >> Sudarshan Jagadale >> >> From: Sudarshan Jagadale/Austin/Contr/IBM >> To: Nishidha Panpaliya/Austin/Contr/IBM@IBMUS >> Cc: *[email protected]* <[email protected]>, Manish >> Patil/Austin/Contr/IBM@IBMUS >> Date: 03/18/2016 11:04 AM >> Subject: Fw: Debugging Impala code >> ------------------------------ >> >> >> >> Thanks and Regards >> Sudarshan Jagadale >> Power Open Source Solutions >> ----- Forwarded by Sudarshan Jagadale/Austin/Contr/IBM on 03/18/2016 >> 11:04 AM ----- >> >> From: Tim Armstrong <*[email protected]* >> <[email protected]>> >> To: *[email protected]* >> <[email protected]> >> Cc: Sudarshan Jagadale/Austin/Contr/IBM@IBMUS >> Date: 03/17/2016 10:39 PM >> Subject: Re: Debugging Impala code >> ------------------------------ >> >> >> >> Was it the impalad process that crashed? If so, there are a few >> places you can check: >> - Look in /tmp/impalad.ERROR, /tmp/impalad_node1.ERROR and >> /tmp/impalad_node2.ERROR for error messages. If it hit an >> assertion, you >> will get the message in there. >> - Look in the equivalent INFO logs for other error >> messages (for some crashes, there is info sent to INFO but >> not ERROR) >> - Look for hs_err_pid*.log files in the directory you ran >> Impala from. These are crash reports from the embedded JVM in >> the impalad >> process >> - Get impala to produce a core dump (make sure you have >> ulimit -c unlimited set when starting the cluster. I have it >> set in my >> .bashrc file) then debug with gdb. >> >> >> On Thu, Mar 17, 2016 at 8:59 AM, Jim Apple <*[email protected]* >> <[email protected]>> wrote: >> I believe Hive is sometimes used for data loading, though I'm not >> sure. >> >> I haven't debugged impala during data loading, but when I do >> need to debug >> the backend, I often do >> >> sudo gdb -p $(ps -C impalad -o pid | tail -1 | awk '{print $1}') >> >> >> On Thu, Mar 17, 2016 at 8:50 AM, Nishidha Panpaliya < >> *[email protected]* <[email protected]>> >> wrote: >> >> > >> > Hi All, >> > >> > I'm able to build Impala on Ubuntu ppc64le but getting >> crashes while >> > loading test data. >> > >> > I wanted to know how do you normally debug Impala code while >> loading test >> > data before running unit tests. Other than core dump, what >> are the other >> > ways to find out causes of crash in Impala? >> > >> > >> > Thanks, >> > Nishidha >> > >> >> >> >> >> >
