I agree with Jim's suggestion. On Tue, Apr 5, 2016 at 8:59 AM, Jim Apple <[email protected]> wrote:
> Sudarshan, you have asked, "Please help on list of test issues which > Nishidha has mentioned." > > In my personal opinion, it would help to tackle each issue separately, in > a separate thread or JIRA for each. How does that sound? > > On Mon, Apr 4, 2016 at 1:15 AM, Sudarshan Jagadale <[email protected]> > wrote: > >> Dear Jim, Silvius and Team, >> >> Its be great to work closely with Impala community, we have been very >> satisfied with all help and collaborative working with you. >> Last Thursday we had call with Silvius and had very much useful call, in >> near future we will be having another call with David, IBM Power solution >> Arch will be joining. >> >> We have some blockers which Nishidha has mentioned in email, We are also >> working on them but we appreciate if could get some help out from dev team. >> >> We will be share another patch, for which we have legal approval. >> >> Please help on list of test issues which Nishidha has mentioned. >> >> >> Thanks and Regards >> Sudarshan Jagadale >> Power Open Source Solutions >> >> [image: Inactive hide details for Nishidha Panpaliya---04/01/2016 >> 06:36:24 PM---Sorry, attachment size was too big. So, here is the >> com]Nishidha >> Panpaliya---04/01/2016 06:36:24 PM---Sorry, attachment size was too big. >> So, here is the compressed log and backend test analysis before >> >> From: Nishidha Panpaliya/Austin/Contr/IBM >> To: "Tim Armstrong" <[email protected]>, "Jim Apple" < >> [email protected]>, [email protected] >> Cc: "Silvius Rus" <[email protected]>, Sudarshan >> Jagadale/Austin/Contr/IBM@IBMUS >> Date: 04/01/2016 06:36 PM >> Subject: Fw: Debugging Impala code >> ------------------------------ >> >> >> Sorry, attachment size was too big. So, here is the compressed log and >> backend test analysis before and after rebase. >> [attachment "LastTest.zip" deleted by Sudarshan >> Jagadale/Austin/Contr/IBM] >> [attachment "BackendTestAnalysis.xls" deleted by Sudarshan >> Jagadale/Austin/Contr/IBM] >> >> Thanks, >> Nishidha >> >> ----- Forwarded by Nishidha Panpaliya/Austin/Contr/IBM on 04/01/2016 >> 06:34 PM ----- >> >> From: Nishidha Panpaliya/Austin/Contr/IBM >> To: [email protected] >> Date: 04/01/2016 06:33 PM >> Subject: Fw: Debugging Impala code >> ------------------------------ >> >> >> Resending again, as it failed to deliver first time. >> >> ----- Forwarded by Nishidha Panpaliya/Austin/Contr/IBM on 04/01/2016 >> 06:32 PM ----- >> >> From: Nishidha Panpaliya/Austin/Contr/IBM >> To: Jim Apple <[email protected]>, Tim Armstrong < >> [email protected]> >> Cc: David Clissold/Austin/IBM@IBMUS, [email protected], >> Manish Patil/Austin/Contr/IBM@IBMUS, Sudarshan >> Jagadale/Austin/Contr/IBM@IBMUS, Valencia Serrao/Austin/Contr/IBM@IBMUS >> Date: 04/01/2016 04:28 PM >> Subject: Re: Fw: Debugging Impala code >> ------------------------------ >> >> >> Hi, >> >> As you've suggested, we've rebased from cdh5-trunk and could compile the >> code on ppc64le with some modifications (ppc specific). >> I also gave a try to backend tests and got the many failures. At first, >> I'd got 41 errors but they all led to a common cause and after fixing it, >> count reduced to 9. >> Please find attached sheet that contains details of test execution before >> and after rebase with their analysis. >> >> [attachment "BackendTestAnalysis.xls" deleted by Nishidha >> Panpaliya/Austin/Contr/IBM] >> >> I've also attached the full log of tests that failed. >> [attachment "LastTest.log" deleted by Nishidha >> Panpaliya/Austin/Contr/IBM] >> >> We are investigating these failures, but in case, you've any pointers, >> we'll be grateful. Also, other than JIRA, is there any place where we can >> see the existing known issues like Jenkins's log? Can we get read-only >> access to those, just to check which failure is on x86 as well and its >> history like which commit introduced it, etc.? >> >> Thanks, >> Nishidha >> >> >> [image: Inactive hide details for Jim Apple ---03/25/2016 10:21:03 >> PM---You can stay on cdh5-trunk. On Fri, Mar 25, 2016 at 9:43 AM, Ni]Jim >> Apple ---03/25/2016 10:21:03 PM---You can stay on cdh5-trunk. On Fri, Mar >> 25, 2016 at 9:43 AM, Nishidha Panpaliya <[email protected] >> >> From: Jim Apple <[email protected]> >> To: Nishidha Panpaliya/Austin/Contr/IBM@IBMUS >> Cc: Tim Armstrong <[email protected]>, David >> Clissold/Austin/IBM@IBMUS, [email protected], Manish >> Patil/Austin/Contr/IBM@IBMUS, Sudarshan Jagadale/Austin/Contr/IBM@IBMUS >> Date: 03/25/2016 10:21 PM >> Subject: Re: Fw: Debugging Impala code >> ------------------------------ >> >> >> >> You can stay on cdh5-trunk. >> >> On Fri, Mar 25, 2016 at 9:43 AM, Nishidha Panpaliya < >> *[email protected]* <[email protected]>> wrote: >> >> Yes, I've started rebasing on cdh5-trunk. As expected, lot of changes >> and conflicts! Is there any release tag or version to which should I >> update >> from cdh5-trunk? >> >> Will update you the build status after merge. >> >> Thanks & Regards, >> Nishidha >> >> [image: Inactive hide details for Tim Armstrong ---03/24/2016 >> 08:08:06 AM---If you haven't already, I'd suggest rebasing on >> cdh5-trunk]Tim >> Armstrong ---03/24/2016 08:08:06 AM---If you haven't already, I'd suggest >> rebasing on cdh5-trunk and taking a look at my patch here that f >> >> From: Tim Armstrong <*[email protected]* >> <[email protected]>> >> To: Jim Apple <*[email protected]* <[email protected]>> >> Cc: Nishidha Panpaliya/Austin/Contr/IBM@IBMUS, David >> Clissold/Austin/IBM@IBMUS, *[email protected]* >> <[email protected]>, Manish Patil/Austin/Contr/IBM@IBMUS, >> Sudarshan Jagadale/Austin/Contr/IBM@IBMUS >> Date: 03/24/2016 08:08 AM >> Subject: Re: Fw: Debugging Impala code >> ------------------------------ >> >> >> >> If you haven't already, I'd suggest rebasing on cdh5-trunk and taking >> a look at my patch here that fixes those tests for later versions of LLVM >> (on x86): *http://gerrit.cloudera.org/#/c/2486/* >> <http://gerrit.cloudera.org/#/c/2486/> . There were a lot of subtle >> issues so it will save you a lot of time. >> >> - Tim >> >> On Wed, Mar 23, 2016 at 9:45 AM, Jim Apple <*[email protected]* >> <[email protected]>> wrote: >> You might try looking for your broken tests in the bug tracker. >> For instance, looking for expr-test leads to >> *https://issues.cloudera.org/browse/IMPALA-2995* >> <https://issues.cloudera.org/browse/IMPALA-2995>. >> >> On Wed, Mar 23, 2016 at 2:33 AM, Nishidha Panpaliya < >> *[email protected]* <[email protected]>> wrote: >> Hi Tim and Jim, >> >> Once again I thank you for your quick help. >> >> I ran run-backend-tests.sh and here is the result of all tests - >> 89% tests passed, 8 tests failed out of 71 >> >> Total Test time (real) = 979.08 sec >> >> The following tests FAILED: >> 1 - llvm-codegen-test (SEGFAULT) >> 13 - expr-test (OTHER_FAULT) >> 14 - expr-codegen-test (OTHER_FAULT) >> 19 - data-stream-test (Failed) >> 22 - buffered-block-mgr-test (Failed) >> 32 - tmp-file-mgr-test (Failed) >> 33 - row-batch-serialize-test (SEGFAULT) >> 68 - filesystem-util-test (Failed) >> >> PFA full log. >> * (See attached file: LastTest.log)* >> >> I also investigated some of the failures like >> - *In filesystem-util-test* : Cause of this failure is user >> running the tests being "root". I could not run these >> tests with non-root >> user but I tested a sample test application that does >> exactly same thing as >> done in Createdirectory test of filesystem-util-test >> and it passed with >> non-root user. Also verified this by linux commands >> mkdir, chmod, rmdir >> with root and non-root user. >> - *In tmp-file-mgr-test *: Even this failure looks >> same as it also does chmod and the tries allocating >> space. Since I'm >> running these tests using root user, I would not get >> failure in accessing >> the dir/file even after removing write permissions if >> the user is root. >> - *In* *llvm-codegen-test: *I tried debugging this >> test and found a crash in llvm::Type::getVoidTy(...). >> I'll keep on investigating other crashes and >> segmentation faults. But in case, you find any of the failures >> familiar or >> existing on x86 platforms, please let us know. >> >> Another news is we have got approval to share our patches. >> Soon, I'll be uploading a patch with LLVM up-gradation work. >> >> Thanks, >> Nishidha >> >> >> Thanks, >> Nishidha >> Tim Armstrong ---03/21/2016 09:10:46 PM---We generally run the >> full test suite on machines with at least 32GB of memory: it's >> pretty >> memory hu >> >> From: Tim Armstrong <*[email protected]* >> <[email protected]>> >> To: Jim Apple <*[email protected]* <[email protected]>> >> Cc: *[email protected]* >> <[email protected]>, Sudarshan >> Jagadale/Austin/Contr/IBM@IBMUS, Manish >> Patil/Austin/Contr/IBM@IBMUS, David Clissold/Austin/IBM@IBMUS, >> Nishidha Panpaliya/Austin/Contr/IBM@IBMUS >> Date: 03/21/2016 09:10 PM >> Subject: Re: Fw: Debugging Impala code >> ------------------------------ >> >> >> >> We generally run the full test suite on machines with at least >> 32GB of memory: it's pretty memory hungry because you have 3 >> Impalads >> running side-by-side. I believe we tend to run the full data load on >> machines with even more memory. You can start the test cluster with >> a >> single impalad before running tests (./bin/start-impala-cluster -s1 >> && >> ./tests/run-tests.py). Some tests will fail since they assume 3 >> Impalads >> but most should work ok. >> >> Starting with the backend tests sounds like a good idea - they >> do exercise some of the codegen and other architecture-dependent >> parts that >> will likely be tricky. >> >> - Tim >> >> On Mon, Mar 21, 2016 at 5:09 AM, Jim Apple < >> *[email protected]* <[email protected]>> wrote: >> I think you should be able to run the backend tests without >> data loading: >> >> ./bin/run-backend-tests.sh >> # or >> ctest >> >> As in the frontend tests, you can specify which >> test you want to run: >> >> ctest --output-on-failure -R expr-test # also shows >> what broke, if anything >> >> To only build the backend test run: >> >> make be-test >> >> On Mon, Mar 21, 2016 at 4:12 AM, Nishidha Panpaliya >> <*[email protected]* <[email protected]>> wrote: >> Thanks Jim and Tim for your replies. Really >> appreciate your co-operation and promptness. >> >> I've a few more queries - >> >> 1. What is the memory requirement of Impala to run >> all the tests? Currently, I see test data creation and >> loading is consuming >> almost 7GB of RAM. And after this, it gets stopped with >> bad_alloc >> exception. I've already requested to increase RAM of my >> VM. But just wanted >> to know if 16GB will suffice. >> >> 2. Can we skip load testing at this stage and >> simply run basic unit tests at first? Or is there any >> setting by means of >> which we can lower the volume of test data being >> generated/loaded? Once >> basic tests are working, we can focus on load testing. >> >> Also, we wish to have a call with you to discuss >> all this. We are located in India. >> >> Thanks, >> Nishidha >> >> >> Sudarshan Jagadale---03/18/2016 11:04:45 >> AM---Thanks and Regards Sudarshan Jagadale >> >> From: Sudarshan Jagadale/Austin/Contr/IBM >> To: Nishidha Panpaliya/Austin/Contr/IBM@IBMUS >> Cc: *[email protected]* <[email protected]>, >> Manish Patil/Austin/Contr/IBM@IBMUS >> Date: 03/18/2016 11:04 AM >> Subject: Fw: Debugging Impala code >> ------------------------------ >> >> >> >> Thanks and Regards >> Sudarshan Jagadale >> Power Open Source Solutions >> ----- Forwarded by Sudarshan >> Jagadale/Austin/Contr/IBM on 03/18/2016 11:04 AM ----- >> >> From: Tim Armstrong <*[email protected]* >> <[email protected]>> >> To: *[email protected]* >> <[email protected]> >> Cc: Sudarshan Jagadale/Austin/Contr/IBM@IBMUS >> Date: 03/17/2016 10:39 PM >> Subject: Re: Debugging Impala code >> ------------------------------ >> >> >> >> Was it the impalad process that crashed? If so, >> there are a few places you can check: >> - Look in /tmp/impalad.ERROR, >> >> /tmp/impalad_node1.ERROR and /tmp/impalad_node2.ERROR for error messages. >> If it >> hit an assertion, you will get the message in there. >> - Look >> in the >> equivalent INFO logs for other error messages (for some crashes, >> there >> is info sent to INFO but not ERROR) >> - Look >> for >> hs_err_pid*.log files in the directory you ran Impala from. These are >> crash >> reports from the embedded JVM in the impalad process >> - Get >> impala >> to produce a core dump (make sure you have ulimit -c unlimited set >> when >> starting the cluster. I have it set in my .bashrc file) then debug >> with >> gdb. >> >> >> On Thu, Mar 17, 2016 at 8:59 AM, Jim Apple < >> *[email protected]* <[email protected]>> >> wrote: >> I believe Hive is sometimes used for data >> loading, though I'm not sure. >> >> I haven't debugged impala >> during data loading, but when I >> do need to debug >> the backend, I often do >> >> sudo gdb -p $(ps -C impalad >> -o pid | tail -1 | awk '{print >> $1}') >> >> >> On Thu, Mar 17, 2016 at >> 8:50 AM, Nishidha Panpaliya < >> *[email protected]* >> <[email protected]>> >> wrote: >> >> > >> > Hi All, >> > >> > I'm able to build Impala >> on Ubuntu ppc64le but getting >> crashes while >> > loading test data. >> > >> > I wanted to know how do >> you normally debug Impala code >> while loading test >> > data before running unit >> tests. Other than core dump, >> what are the other >> > ways to find out causes >> of crash in Impala? >> > >> > >> > Thanks, >> > Nishidha >> > >> >> >> >> >> >
