Ahh, thanks Casey. Did not know about that. Valencia, Impala's data loading expects the files to be placed in IMPALA_HOME/testdata/impala-data
On Tue, May 3, 2016 at 11:21 PM, Casey Ching <[email protected]> wrote: > Comment inline below > > > On May 3, 2016 at 11:18:06 PM, Alex Behm ([email protected]) wrote: > > Hi Valencia, > > I'm sorry you are having so much trouble with our setup. Let's see what we > can do. > > There was an infra issue with receiving the logs you sent me. The > email/attachment got rejected on our side. Maybe you can upload the logs > somewhere so I can grab them? > > See more responses inline below. > > On Sat, Apr 30, 2016 at 5:01 AM, Valencia Serrao <[email protected]> > wrote: > > > Hi Alex, > > > > I was going more deeper through the logs. I have some findings and > queries: > > > > 1. At the "Invalidating Metadata" step (as mentioned in below mail), i > > noticed that, it is trying to use kerberos. Perhaps, this is preventing > the > > testdata generation from proceeding, as we are not using Kerberos. > > I need to know how this can be done without involving Kerberos support ? > > > Kerberos is certainly not needed to build and run tests. > > > > > 2. I had executed the fe tests despite the incomplete testdata > generation, > > the tests started and surely have failed. Many of these (null pointer > > exception in AuthorzationTests) have a common cause: "tpch database does > > not exist." > > e.g. as shown in .Impala/cluster_logs/query_tests/test-run-workload.log. > > > > Does the "tpch" database gets created after the current blocker step > > "Invalidating Metadata" ? > > > > Yes, the TPCH database is created and loaded as part of that first phase. > However, the data files are not yet publicly accessible. Let me work on > that from my side, and get back to you soon. One way or the other we'll be > able to provide you with the data. > > > The data is at > https://github.com/cloudera/Impala-docker-hub/tree/master/prereqs/container_root/tmp > . > The files are split into 50 MB pieces for git. You can put them back > together as is done in > https://github.com/cloudera/Impala-docker-hub/blob/master/complete/Dockerfile > > > > > > 3. In the fe test console output log, another error shown: > > ============================= test session starts > > ============================== > > platform linux2 -- Python 2.7.5 -- py-1.4.30 -- pytest-2.7.2 > > rootdir: /work/, inifile: > > plugins: random, xdist > > ERROR: file not found:/work/I > > mpala/../Impala-auxiliary-tests/tests/aux_custom_cluster_tests/ > > > > These are not present/created on my vm. May i know when these get > created ? > > > > 4. Could you also share the total number of fe tests ? > > > > I'll privately send you the console output from a successful FE run. > Hopefully that can help. > > Cheers, > > Alex > > > > > > > Looking forward to your reply. > > > > Regards, > > Valencia > > > > > > [image: Inactive hide details for Valencia Serrao---04/30/2016 09:05:54 > > AM---Hi Alex, I've been able to make some progress on testdata]Valencia > > Serrao---04/30/2016 09:05:54 AM---Hi Alex, I've been able to make some > > progress on testdata generation, however, i still face the foll > > > > From: Valencia Serrao/Austin/Contr/IBM > > To: [email protected], Alex Behm <[email protected]> > > Cc: Sudarshan Jagadale/Austin/Contr/IBM@IBMUS, Nishidha > > Panpaliya/Austin/Contr/IBM@IBMUS, Valencia Serrao/Austin/Contr/IBM@IBMUS > > Date: 04/30/2016 09:05 AM > > Subject: Fw: Issues with generating testdata for Impala > > ------------------------------ > > > > > > > > Hi Alex, > > > > I've been able to make some progress on testdata generation, however, i > > still face the following issues: > > > > > > > ******************************************************************************************************************************************************************* > > > Invalidating Metadata > > > > > (load-functional-query-exhaustive-impala-load-generated-parquet-none-none.sql): > > > INSERT OVERWRITE TABLE functional_parquet.alltypes partition (year, > month) > > SELECT id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, > > float_col, double_col, date_string_col, string_col, timestamp_col, year, > > month > > FROM functional.alltypes > > > > Data Loading from Impala failed with error: ImpalaBeeswaxException: > > INNER EXCEPTION: <class 'socket.error'> > > MESSAGE: [Errno 104] Connection reset by peer > > Error in /root/nishidha/Impala/testdata/bin/create-load-data.sh at line > > 41: while [ -n "$*" ] > > Error in /root/nishidha/Impala/buildall.sh at line 368: > > ${IMPALA_HOME}/testdata/bin/create-load-data.sh ${CREATE_LOAD_DATA_ARGS} > > <<< Y > > > > > ************************************************************************************************************************************************************************* > > > > > i continued with fe tests as is. Here is the complete output log. > > [attachment "fe_test_output.zip" deleted by Valencia > > Serrao/Austin/Contr/IBM] > > > > Cluster logs: [attachment "cluster_logs.7z" deleted by Valencia > > Serrao/Austin/Contr/IBM] > > > > Kindly guide me on the same. > > > > Regards, > > Valencia > > ----- Forwarded by Valencia Serrao/Austin/Contr/IBM on 04/29/2016 10:57 > AM > > ----- > > > > From: Sudarshan Jagadale/Austin/Contr/IBM > > To: Valencia Serrao/Austin/Contr/IBM@IBMUS > > Date: 04/29/2016 10:49 AM > > Subject: Fw: Issues with generating testdata for Impala > > ------------------------------ > > > > > > FYI > > Thanks and Regards > > Sudarshan Jagadale > > Power Open Source Solutions > > ----- Forwarded by Sudarshan Jagadale/Austin/Contr/IBM on 04/29/2016 > 10:48 > > AM ----- > > > > From: Alex Behm <[email protected]> > > To: [email protected] > > Cc: Sudarshan Jagadale/Austin/Contr/IBM@IBMUS, Nishidha > > Panpaliya/Austin/Contr/IBM@IBMUS > > Date: 04/28/2016 09:34 PM > > Subject: Re: Issues with generating testdata for Impala > > ------------------------------ > > > > > > > > Hi Valencia, > > > > sorry I did not get the attachment. Would you be able to tar.gz and > attach > > the whole cluster_logs directory? > > > > Alex > > > > On Thu, Apr 28, 2016 at 6:23 AM, Valencia Serrao <*[email protected]* > > <[email protected]>> wrote: > > > > Hi Alex, > > > > I tried building impala again with the following: > > HDFS CDH 5.7.0 ( > > * > http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_57.html#topic_3* > > < > http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_57.html#topic_3> > > > ) > > HBASE CDH 5.7.0 SNAPSHOT ( > > *http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.7.0.tar.gz* > > <http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.7.0.tar.gz> ) > > - this required to patch in a fix ( > > * > https://issues.apache.org/jira/secure/attachment/12792536/HBASE-15322-branch-1.2.patch* > > < > https://issues.apache.org/jira/secure/attachment/12792536/HBASE-15322-branch-1.2.patch> > > > ) > > HIVE CDH 5.8.0 SNAPSHOT > > > > With the above combination, i'm able to move past the exception and > > also have the RegionServer service up and running. However, it now gives > > error as below: > > > > > > > ******************************************************************************************************************** > > > (load-functional-query-exhaustive-impala-generated-text-none-none.sql): > > CREATE EXTERNAL TABLE IF NOT EXISTS functional.decimal_tbl ( > > d1 DECIMAL, > > d2 DECIMAL(10, 0), > > d3 DECIMAL(20, 10), > > d4 DECIMAL(38, 38), > > d5 DECIMAL(10, 5)) > > PARTITIONED BY (d6 DECIMAL(9, 0)) > > ROW FORMAT delimited fields terminated by ',' > > STORED AS TEXTFILE > > LOCATION '/test-warehouse/decimal_tbl' > > > > (load-functional-query-exhaustive-impala-generated-text-none-none.sql): > > USE functional > > > > (load-functional-query-exhaustive-impala-generated-text-none-none.sql): > > ALTER TABLE decimal_tbl ADD IF NOT EXISTS PARTITION(d6=1) > > > > Data Loading from Impala failed with error: ImpalaBeeswaxException: > > INNER EXCEPTION: <class > > 'impala._thrift_gen.beeswax.ttypes.BeeswaxException'> > > MESSAGE: > > Error: null > > > > > ****************************************************************************************************************** > > > > > Here is the complete log for the same. *(See attached file: > > data-load-functional-exhaustive.log)* > > > > It would great if you could guide me on this issue, so i could proceed > > with the fe tests. > > > > Still awaiting link to the source code of HDFS CDH 5.8.0 > > > > Regards, > > Valencia > > > > > > > > > >
