Hi Nishidha, I started working on our next release cycle towards the end of last week, so I've been looking at LLVM 3.7 and have made a bit of progress getting it working on intel. We're trying to get it done working so we have plenty of chance to test it.
RE the TTransportException error, that is often because of a crash. Usually to debug I would first look at the /tmp/impalad.ERROR and /tmp/impalad.INFO logs for the cause of the crash. The embedded JVM also generates hs_err_pid*.log files with a crash report that can sometimes be useful. If that doesn't reveal the cause, then I'd look to see if there is a core dump in the Impala directory (I normally run with "ulimit -c unlimited" set so that a crash will generate a core file). I already fixed a couple of problems with codegen in LLVM 3.7, including one crash that was an assertion about struct sizes. I'll be posting the patch soon once I've done a bit more testing. It might help to make progress is you disable LLVM codegen by default during data loading by setting the following environment variable: export START_CLUSTER_ARGS='--impalad_args=-default_query_options="disable_codegen=1"' You can also start the test cluster with the same arguments or just set it in the set with "set disable_codegen=1). ./bin/start-impala-cluster.py --impalad_args=-default_query_options="disable_codegen=1" On Mon, Mar 7, 2016 at 5:13 AM, Nishidha Panpaliya <[email protected]> wrote: > Hi Tim, > > Yes, I could fix this snappyError by building snappy-java for Power and > adding the native library for power into existing snappy-java-1.0.4.1.jar > used by hbase, hive, sentry and hadoop. > The test data loading has been proceeded further and gave a new exception > which I'm looking into and as below. > > Data Loading from Impala failed with error: ImpalaBeeswaxException: > INNER EXCEPTION: <class 'thrift.transport.TTransport.TTransportException'> > MESSAGE: None > > Also, I've been able to start impala and try just one following query as > given in https://github.com/cloudera/Impala/wiki/How-to-build-Impala- > impala-shell.sh -q"SELECT version()" > > And regarding patch of my work, I'm sorry for the delay. Although it does > not need any CLA to be signed, but it is under discussion with our IBM > legal team, just to ensure we are compliant with the policies. Hoping to > update you on this soon. Could you tell me when are you going to start with > this new release cycle? > > Thanks, > Nishidha > > [image: Inactive hide details for Tim Armstrong ---03/05/2016 03:14:29 > AM---It also looks like it got far enough that you should have a]Tim > Armstrong ---03/05/2016 03:14:29 AM---It also looks like it got far enough > that you should have a bit of data loaded - have you been able > > From: Tim Armstrong <[email protected]> > To: nishidha panpaliya <[email protected]> > Cc: Impala Dev <[email protected]>, Nishidha > Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan > Jagadale/Austin/Contr/IBM@IBMUS > Date: 03/05/2016 03:14 AM > Subject: Re: LeaseExpiredException while test data creation on ppc64le > ------------------------------ > > > > It also looks like it got far enough that you should have a bit of data > loaded - have you been able to start impala and run queries on some of > those tables? > > We're starting a new release cycle so I'm actually about to focus on > upgrading our version of LLVM to 3.7 and getting the Intel support working. > I think we're going to be putting a bit of effort into reducing LLVM code > generation time: it seems like LLVM 3.7 is slightly slower in some cases. > > We should stay in sync, it would be good to make sure that any changes I > make will work for your PowerPC work too. If you want to share any patches > (even if you're not formally contributing them) it would be helpful for me > to understand what you have already done on this path. > > Cheers, > Tim > > On Fri, Mar 4, 2016 at 1:40 PM, Tim Armstrong <*[email protected]* > <[email protected]>> wrote: > > > Hi Nishidha, > It looks like Hive is maybe missing the native snappy library: I see > this in the logs: > > java.lang.Exception: org.xerial.snappy.SnappyError: > [FAILED_TO_LOAD_NATIVE_LIBRARY] null > at > > org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) > Caused by: org.xerial.snappy.SnappyError: > [FAILED_TO_LOAD_NATIVE_LIBRARY] null > at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229) > at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44) > at org.apache.avro.file.SnappyCodec.compress(SnappyCodec.java:43) > at > > org.apache.avro.file.DataFileStream$DataBlock.compressUsing(DataFileStream.java:361) > at > org.apache.avro.file.DataFileWriter.writeBlock(DataFileWriter.java:394) > at > org.apache.avro.file.DataFileWriter.sync(DataFileWriter.java:413) > > > > If you want to try making progress without Hive snappy support, I > think you coudl disable some of the files formats by editing > testdata/workloads/*/*.csv and removing some of the "snap" file formats. > The impala test suite generates data in many different file formats with > different compression settings. > > > On Wed, Mar 2, 2016 at 7:08 AM, nishidha panpaliya < > *[email protected]* <[email protected]>> wrote: > Hello, > > After building Impala on ppc64le, I'm trying to run all the tests of > Impala. In the process, I'm getting an error while test data creation. > Command ran - > ${IMPALA_HOME}/buildall.sh -testdata -format > Output - Attached log (output.txt) > > Also attached logs named > cluster_logs/data_loading/data-load-functional-exhaustive.log. And > hive.log. > > I tried setting below parameters in hive-site.xml but of no use. > hive.exec.max.dynamic.partitions=100000; > hive.exec.max.dynamic.partitions.pernode=100000; > hive.exec.parallel=false > > I'll be really thankful if you could provide me some help here. > > Thanks in advance, > Nishidha > > -- > You received this message because you are subscribed to the Google > Groups "Impala Dev" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to *[email protected]* > <[email protected]>. > > > >
