Re: test errors in local env

[Amos Bird]

Java 7 solves this problem. Thanks Lars!

Lars Volker writes:

> Hi Amos,
>
> apologies for the delay? Is this issue still open? If so, can you try to
> run the tests with Java 7?
>
> Thanks, Lars
>
> On Thu, Sep 22, 2016 at 6:38 PM, Amos Bird  wrote:
>
>> Thanks Lars,
>>
>> My Java version is
>>
>> $ $JAVA_HOME/bin/java -version
>> java version "1.8.0_65"
>> Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
>> Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
>>
>> > Hi Amos,
>> >
>> > I tried to see if IMPALA-3643
>> >  is still reproducible
>> for
>> > me. In the process I also ran into the "NoClassDefFoundError" error you
>> saw
>> > and found out, that this happens if my local hadoop services
>> > (./testdata/bin/run-all.sh) isn't running.
>> >
>> > Once I have the local hadoop services running I'm able to repro the issue
>> > in IMPALA-3643. The only thing I remember doing differently from the
>> > recommended ways of dev-machine setup is using Java 8. The Java 8 docs on
>> > LinkedHashSet
>> > 
>> don't
>> > read like they'd change the ordering, however it doesn't say that the
>> > ordering has to be stable *between different versions of Java*. I will
>> try
>> > to confirm that this issue does not repro with Java 7 when I have time.
>> Can
>> > you check the Java version you're running. Mine is this:
>> >
>> > $ $JAVA_HOME/bin/java -version
>> > java version "1.8.0_101"
>> > Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
>> > Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
>> >
>> > Alex, could a difference in the implementation between Java 7 and 8 be a
>> > reasonable explanation for this?
>> >
>> > Cheers, Lars
>> >
>> > On Thu, Sep 22, 2016 at 12:52 AM, Amos Bird  wrote:
>> >
>> >>
>> >> Thank you for helping me out :D
>> >>
>> >> > Thanks for keeping this discussion going.
>> >> >
>> >> > On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird 
>> wrote:
>> >> >
>> >> >>
>> >> >> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird 
>> >> wrote:
>> >> >> >
>> >> >> >>
>> >> >> >> Hi there,
>> >> >> >>
>> >> >> >> I followed the wiki
>> >> >> >> https://cwiki.apache.org/confluence/display/IMPALA/How+
>> >> >> >> to+load+and+run+Impala+tests
>> >> >> >> carefully but still have some problems in my local env.
>> >> >> >>
>> >> >> >> 1. I need to manually execute "hdfs dfs -mkdir
>> >> >> /test-warehouse/emptytable"
>> >> >> >> to get rid of some fe test error.
>> >> >> >>
>> >> >> >>
>> >> >> > Ideally, you should not have to do this. Could you tell me what
>> errors
>> >> >> you
>> >> >> > encountered? Sounds like there may be a test or data loading bug we
>> >> >> should
>> >> >> > fix.
>> >> >>
>> >> >> The error is :
>> >> >>
>> >> >> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest)  Time
>> >> >> elapsed: 0.033 sec  <<< FAILURE!
>> >> >> java.lang.AssertionError: got error:
>> >> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
>> does
>> >> >> not exist.
>> >> >> expected:
>> >> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
>> >> >> contains no visible files.
>> >> >>   at org.junit.Assert.fail(Assert.java:88)
>> >> >>   at org.junit.Assert.assertTrue(Assert.java:41)
>> >> >>   at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
>> >> >> FrontendTestBase.java:312)
>> >> >>   at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
>> >> >> FrontendTestBase.java:292)
>> >> >>   at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(
>> >> >> AnalyzeStmtsTest.java:2860)
>> >> >>
>> >> >>
>> >> > Do you have a table functional.emptytable? If yes, then what location
>> is
>> >> > reported in "show create table"?
>> >> Query: show create table functional.emptytable
>> >> +-+
>> >> | result  |
>> >> +-+
>> >> | CREATE EXTERNAL TABLE functional.emptytable (   |
>> >> |   field STRING  |
>> >> | )   |
>> >> | PARTITIONED BY (|
>> >> |   f2 INT|
>> >> | )   |
>> >> | STORED AS TEXTFILE  |
>> >> | LOCATION 'hdfs://localhost:20500/test-warehouse/emptytable' |
>> >> | TBLPROPERTIES ('transient_lastDdlTime'='1464782625')|
>> >> +-+
>> >> Fetched 1 row(s) in 5.51s
>> >>
>> >> > Does the directory exist in HDFS?
>> >> No.
>> >>
>> >> >
>> >> > You could try to manually reload the table and see if the directory is
>> >> > created:
>> >> > bin/load-data.py -f

Re: test errors in local env

[Lars Volker]

Hi Amos,

apologies for the delay? Is this issue still open? If so, can you try to
run the tests with Java 7?

Thanks, Lars

On Thu, Sep 22, 2016 at 6:38 PM, Amos Bird  wrote:

> Thanks Lars,
>
> My Java version is
>
> $ $JAVA_HOME/bin/java -version
> java version "1.8.0_65"
> Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
>
> > Hi Amos,
> >
> > I tried to see if IMPALA-3643
> >  is still reproducible
> for
> > me. In the process I also ran into the "NoClassDefFoundError" error you
> saw
> > and found out, that this happens if my local hadoop services
> > (./testdata/bin/run-all.sh) isn't running.
> >
> > Once I have the local hadoop services running I'm able to repro the issue
> > in IMPALA-3643. The only thing I remember doing differently from the
> > recommended ways of dev-machine setup is using Java 8. The Java 8 docs on
> > LinkedHashSet
> > 
> don't
> > read like they'd change the ordering, however it doesn't say that the
> > ordering has to be stable *between different versions of Java*. I will
> try
> > to confirm that this issue does not repro with Java 7 when I have time.
> Can
> > you check the Java version you're running. Mine is this:
> >
> > $ $JAVA_HOME/bin/java -version
> > java version "1.8.0_101"
> > Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
> > Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
> >
> > Alex, could a difference in the implementation between Java 7 and 8 be a
> > reasonable explanation for this?
> >
> > Cheers, Lars
> >
> > On Thu, Sep 22, 2016 at 12:52 AM, Amos Bird  wrote:
> >
> >>
> >> Thank you for helping me out :D
> >>
> >> > Thanks for keeping this discussion going.
> >> >
> >> > On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird 
> wrote:
> >> >
> >> >>
> >> >> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird 
> >> wrote:
> >> >> >
> >> >> >>
> >> >> >> Hi there,
> >> >> >>
> >> >> >> I followed the wiki
> >> >> >> https://cwiki.apache.org/confluence/display/IMPALA/How+
> >> >> >> to+load+and+run+Impala+tests
> >> >> >> carefully but still have some problems in my local env.
> >> >> >>
> >> >> >> 1. I need to manually execute "hdfs dfs -mkdir
> >> >> /test-warehouse/emptytable"
> >> >> >> to get rid of some fe test error.
> >> >> >>
> >> >> >>
> >> >> > Ideally, you should not have to do this. Could you tell me what
> errors
> >> >> you
> >> >> > encountered? Sounds like there may be a test or data loading bug we
> >> >> should
> >> >> > fix.
> >> >>
> >> >> The error is :
> >> >>
> >> >> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest)  Time
> >> >> elapsed: 0.033 sec  <<< FAILURE!
> >> >> java.lang.AssertionError: got error:
> >> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
> does
> >> >> not exist.
> >> >> expected:
> >> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
> >> >> contains no visible files.
> >> >>   at org.junit.Assert.fail(Assert.java:88)
> >> >>   at org.junit.Assert.assertTrue(Assert.java:41)
> >> >>   at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
> >> >> FrontendTestBase.java:312)
> >> >>   at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
> >> >> FrontendTestBase.java:292)
> >> >>   at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(
> >> >> AnalyzeStmtsTest.java:2860)
> >> >>
> >> >>
> >> > Do you have a table functional.emptytable? If yes, then what location
> is
> >> > reported in "show create table"?
> >> Query: show create table functional.emptytable
> >> +-+
> >> | result  |
> >> +-+
> >> | CREATE EXTERNAL TABLE functional.emptytable (   |
> >> |   field STRING  |
> >> | )   |
> >> | PARTITIONED BY (|
> >> |   f2 INT|
> >> | )   |
> >> | STORED AS TEXTFILE  |
> >> | LOCATION 'hdfs://localhost:20500/test-warehouse/emptytable' |
> >> | TBLPROPERTIES ('transient_lastDdlTime'='1464782625')|
> >> +-+
> >> Fetched 1 row(s) in 5.51s
> >>
> >> > Does the directory exist in HDFS?
> >> No.
> >>
> >> >
> >> > You could try to manually reload the table and see if the directory is
> >> > created:
> >> > bin/load-data.py -f -w functional-query --table_names=emptytable
> >> > --table_formats=text/none
> >> After executing this command the directory appears.
> >>
> >> >
> >> >>>
> >> >> >> 2. I have authz-policy.ini in HDF

Re: test errors in local env

[Amos Bird]

Thanks Lars,

My Java version is

$ $JAVA_HOME/bin/java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)

> Hi Amos,
>
> I tried to see if IMPALA-3643
>  is still reproducible for
> me. In the process I also ran into the "NoClassDefFoundError" error you saw
> and found out, that this happens if my local hadoop services
> (./testdata/bin/run-all.sh) isn't running.
>
> Once I have the local hadoop services running I'm able to repro the issue
> in IMPALA-3643. The only thing I remember doing differently from the
> recommended ways of dev-machine setup is using Java 8. The Java 8 docs on
> LinkedHashSet
>  don't
> read like they'd change the ordering, however it doesn't say that the
> ordering has to be stable *between different versions of Java*. I will try
> to confirm that this issue does not repro with Java 7 when I have time. Can
> you check the Java version you're running. Mine is this:
>
> $ $JAVA_HOME/bin/java -version
> java version "1.8.0_101"
> Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
>
> Alex, could a difference in the implementation between Java 7 and 8 be a
> reasonable explanation for this?
>
> Cheers, Lars
>
> On Thu, Sep 22, 2016 at 12:52 AM, Amos Bird  wrote:
>
>>
>> Thank you for helping me out :D
>>
>> > Thanks for keeping this discussion going.
>> >
>> > On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird  wrote:
>> >
>> >>
>> >> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird 
>> wrote:
>> >> >
>> >> >>
>> >> >> Hi there,
>> >> >>
>> >> >> I followed the wiki
>> >> >> https://cwiki.apache.org/confluence/display/IMPALA/How+
>> >> >> to+load+and+run+Impala+tests
>> >> >> carefully but still have some problems in my local env.
>> >> >>
>> >> >> 1. I need to manually execute "hdfs dfs -mkdir
>> >> /test-warehouse/emptytable"
>> >> >> to get rid of some fe test error.
>> >> >>
>> >> >>
>> >> > Ideally, you should not have to do this. Could you tell me what errors
>> >> you
>> >> > encountered? Sounds like there may be a test or data loading bug we
>> >> should
>> >> > fix.
>> >>
>> >> The error is :
>> >>
>> >> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest)  Time
>> >> elapsed: 0.033 sec  <<< FAILURE!
>> >> java.lang.AssertionError: got error:
>> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' does
>> >> not exist.
>> >> expected:
>> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
>> >> contains no visible files.
>> >>   at org.junit.Assert.fail(Assert.java:88)
>> >>   at org.junit.Assert.assertTrue(Assert.java:41)
>> >>   at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
>> >> FrontendTestBase.java:312)
>> >>   at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
>> >> FrontendTestBase.java:292)
>> >>   at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(
>> >> AnalyzeStmtsTest.java:2860)
>> >>
>> >>
>> > Do you have a table functional.emptytable? If yes, then what location is
>> > reported in "show create table"?
>> Query: show create table functional.emptytable
>> +-+
>> | result  |
>> +-+
>> | CREATE EXTERNAL TABLE functional.emptytable (   |
>> |   field STRING  |
>> | )   |
>> | PARTITIONED BY (|
>> |   f2 INT|
>> | )   |
>> | STORED AS TEXTFILE  |
>> | LOCATION 'hdfs://localhost:20500/test-warehouse/emptytable' |
>> | TBLPROPERTIES ('transient_lastDdlTime'='1464782625')|
>> +-+
>> Fetched 1 row(s) in 5.51s
>>
>> > Does the directory exist in HDFS?
>> No.
>>
>> >
>> > You could try to manually reload the table and see if the directory is
>> > created:
>> > bin/load-data.py -f -w functional-query --table_names=emptytable
>> > --table_formats=text/none
>> After executing this command the directory appears.
>>
>> >
>> >>>
>> >> >> 2. I have authz-policy.ini in HDFS, but I still get authorization
>> >> errors.
>> >> >>
>> >> >> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest)  Time
>> >> >> elapsed: 0.333 sec  <<< FAILURE!
>> >> >> java.lang.AssertionError: got error:
>> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> default.nodb
>> >> >> expected:
>> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> nodb.alltypes
>> >> >>   at

Re: test errors in local env

[Lars Volker]

Hi Amos,

I tried to see if IMPALA-3643
 is still reproducible for
me. In the process I also ran into the "NoClassDefFoundError" error you saw
and found out, that this happens if my local hadoop services
(./testdata/bin/run-all.sh) isn't running.

Once I have the local hadoop services running I'm able to repro the issue
in IMPALA-3643. The only thing I remember doing differently from the
recommended ways of dev-machine setup is using Java 8. The Java 8 docs on
LinkedHashSet
 don't
read like they'd change the ordering, however it doesn't say that the
ordering has to be stable *between different versions of Java*. I will try
to confirm that this issue does not repro with Java 7 when I have time. Can
you check the Java version you're running. Mine is this:

$ $JAVA_HOME/bin/java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

Alex, could a difference in the implementation between Java 7 and 8 be a
reasonable explanation for this?

Cheers, Lars

On Thu, Sep 22, 2016 at 12:52 AM, Amos Bird  wrote:

>
> Thank you for helping me out :D
>
> > Thanks for keeping this discussion going.
> >
> > On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird  wrote:
> >
> >>
> >> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird 
> wrote:
> >> >
> >> >>
> >> >> Hi there,
> >> >>
> >> >> I followed the wiki
> >> >> https://cwiki.apache.org/confluence/display/IMPALA/How+
> >> >> to+load+and+run+Impala+tests
> >> >> carefully but still have some problems in my local env.
> >> >>
> >> >> 1. I need to manually execute "hdfs dfs -mkdir
> >> /test-warehouse/emptytable"
> >> >> to get rid of some fe test error.
> >> >>
> >> >>
> >> > Ideally, you should not have to do this. Could you tell me what errors
> >> you
> >> > encountered? Sounds like there may be a test or data loading bug we
> >> should
> >> > fix.
> >>
> >> The error is :
> >>
> >> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest)  Time
> >> elapsed: 0.033 sec  <<< FAILURE!
> >> java.lang.AssertionError: got error:
> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' does
> >> not exist.
> >> expected:
> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
> >> contains no visible files.
> >>   at org.junit.Assert.fail(Assert.java:88)
> >>   at org.junit.Assert.assertTrue(Assert.java:41)
> >>   at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
> >> FrontendTestBase.java:312)
> >>   at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
> >> FrontendTestBase.java:292)
> >>   at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(
> >> AnalyzeStmtsTest.java:2860)
> >>
> >>
> > Do you have a table functional.emptytable? If yes, then what location is
> > reported in "show create table"?
> Query: show create table functional.emptytable
> +-+
> | result  |
> +-+
> | CREATE EXTERNAL TABLE functional.emptytable (   |
> |   field STRING  |
> | )   |
> | PARTITIONED BY (|
> |   f2 INT|
> | )   |
> | STORED AS TEXTFILE  |
> | LOCATION 'hdfs://localhost:20500/test-warehouse/emptytable' |
> | TBLPROPERTIES ('transient_lastDdlTime'='1464782625')|
> +-+
> Fetched 1 row(s) in 5.51s
>
> > Does the directory exist in HDFS?
> No.
>
> >
> > You could try to manually reload the table and see if the directory is
> > created:
> > bin/load-data.py -f -w functional-query --table_names=emptytable
> > --table_formats=text/none
> After executing this command the directory appears.
>
> >
> >>>
> >> >> 2. I have authz-policy.ini in HDFS, but I still get authorization
> >> errors.
> >> >>
> >> >> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest)  Time
> >> >> elapsed: 0.333 sec  <<< FAILURE!
> >> >> java.lang.AssertionError: got error:
> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> default.nodb
> >> >> expected:
> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> nodb.alltypes
> >> >>   at org.junit.Assert.fail(Assert.java:88)
> >> >>   at org.junit.Assert.assertTrue(Assert.java:41)
> >> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> AuthorizationTest.java:2220)
> >> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> AuthorizationTest.java:2203)
> >> >>   at com.cloudera.impala.analy

Re: test errors in local env

[Amos Bird]

Thank you for helping me out :D

> Thanks for keeping this discussion going.
>
> On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird  wrote:
>
>>
>> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird  wrote:
>> >
>> >>
>> >> Hi there,
>> >>
>> >> I followed the wiki
>> >> https://cwiki.apache.org/confluence/display/IMPALA/How+
>> >> to+load+and+run+Impala+tests
>> >> carefully but still have some problems in my local env.
>> >>
>> >> 1. I need to manually execute "hdfs dfs -mkdir
>> /test-warehouse/emptytable"
>> >> to get rid of some fe test error.
>> >>
>> >>
>> > Ideally, you should not have to do this. Could you tell me what errors
>> you
>> > encountered? Sounds like there may be a test or data loading bug we
>> should
>> > fix.
>>
>> The error is :
>>
>> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest)  Time
>> elapsed: 0.033 sec  <<< FAILURE!
>> java.lang.AssertionError: got error:
>> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' does
>> not exist.
>> expected:
>> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
>> contains no visible files.
>>   at org.junit.Assert.fail(Assert.java:88)
>>   at org.junit.Assert.assertTrue(Assert.java:41)
>>   at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
>> FrontendTestBase.java:312)
>>   at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
>> FrontendTestBase.java:292)
>>   at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(
>> AnalyzeStmtsTest.java:2860)
>>
>>
> Do you have a table functional.emptytable? If yes, then what location is
> reported in "show create table"?
Query: show create table functional.emptytable
+-+
| result  |
+-+
| CREATE EXTERNAL TABLE functional.emptytable (   |
|   field STRING  |
| )   |
| PARTITIONED BY (|
|   f2 INT|
| )   |
| STORED AS TEXTFILE  |
| LOCATION 'hdfs://localhost:20500/test-warehouse/emptytable' |
| TBLPROPERTIES ('transient_lastDdlTime'='1464782625')|
+-+
Fetched 1 row(s) in 5.51s

> Does the directory exist in HDFS?
No.

>
> You could try to manually reload the table and see if the directory is
> created:
> bin/load-data.py -f -w functional-query --table_names=emptytable
> --table_formats=text/none
After executing this command the directory appears.

>
>>>
>> >> 2. I have authz-policy.ini in HDFS, but I still get authorization
>> errors.
>> >>
>> >> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest)  Time
>> >> elapsed: 0.333 sec  <<< FAILURE!
>> >> java.lang.AssertionError: got error:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> default.nodb
>> >> expected:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> nodb.alltypes
>> >>   at org.junit.Assert.fail(Assert.java:88)
>> >>   at org.junit.Assert.assertTrue(Assert.java:41)
>> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> AuthorizationTest.java:2220)
>> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> AuthorizationTest.java:2203)
>> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> AuthorizationTest.java:2197)
>> >>   at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
>> >> AuthorizationTest.java:512)
>> >>
>> >> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest)  Time
>> >> elapsed: 0.324 sec  <<< FAILURE!
>> >> java.lang.AssertionError: got error:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> default.nodb
>> >> expected:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> nodb.alltypes
>> >>   at org.junit.Assert.fail(Assert.java:88)
>> >>   at org.junit.Assert.assertTrue(Assert.java:41)
>> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> AuthorizationTest.java:2220)
>> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> AuthorizationTest.java:2203)
>> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> AuthorizationTest.java:2197)
>> >>   at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
>> >> AuthorizationTest.java:512)
>> >>
>> >>
>> >> Results :
>> >>
>> >> Failed tests:
>> >>   AuthorizationTest.TestSelect:512->AuthzError:2197->
>> >> AuthzError:2203->AuthzError:2220 got error:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> default.nodb
>> >> expected:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> nodb.alltypes
>> >>   AuthorizationTest.TestSelect:5

Re: test errors in local env

[Alex Behm]

Thanks for keeping this discussion going.

On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird  wrote:

>
> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird  wrote:
> >
> >>
> >> Hi there,
> >>
> >> I followed the wiki
> >> https://cwiki.apache.org/confluence/display/IMPALA/How+
> >> to+load+and+run+Impala+tests
> >> carefully but still have some problems in my local env.
> >>
> >> 1. I need to manually execute "hdfs dfs -mkdir
> /test-warehouse/emptytable"
> >> to get rid of some fe test error.
> >>
> >>
> > Ideally, you should not have to do this. Could you tell me what errors
> you
> > encountered? Sounds like there may be a test or data loading bug we
> should
> > fix.
>
> The error is :
>
> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest)  Time
> elapsed: 0.033 sec  <<< FAILURE!
> java.lang.AssertionError: got error:
> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' does
> not exist.
> expected:
> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
> contains no visible files.
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
> FrontendTestBase.java:312)
>   at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
> FrontendTestBase.java:292)
>   at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(
> AnalyzeStmtsTest.java:2860)
>
>
Do you have a table functional.emptytable? If yes, then what location is
reported in "show create table"?
Does the directory exist in HDFS?

You could try to manually reload the table and see if the directory is
created:
bin/load-data.py -f -w functional-query --table_names=emptytable
--table_formats=text/none

>>
> >> 2. I have authz-policy.ini in HDFS, but I still get authorization
> errors.
> >>
> >> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest)  Time
> >> elapsed: 0.333 sec  <<< FAILURE!
> >> java.lang.AssertionError: got error:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> default.nodb
> >> expected:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> nodb.alltypes
> >>   at org.junit.Assert.fail(Assert.java:88)
> >>   at org.junit.Assert.assertTrue(Assert.java:41)
> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> AuthorizationTest.java:2220)
> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> AuthorizationTest.java:2203)
> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> AuthorizationTest.java:2197)
> >>   at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
> >> AuthorizationTest.java:512)
> >>
> >> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest)  Time
> >> elapsed: 0.324 sec  <<< FAILURE!
> >> java.lang.AssertionError: got error:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> default.nodb
> >> expected:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> nodb.alltypes
> >>   at org.junit.Assert.fail(Assert.java:88)
> >>   at org.junit.Assert.assertTrue(Assert.java:41)
> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> AuthorizationTest.java:2220)
> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> AuthorizationTest.java:2203)
> >>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> AuthorizationTest.java:2197)
> >>   at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
> >> AuthorizationTest.java:512)
> >>
> >>
> >> Results :
> >>
> >> Failed tests:
> >>   AuthorizationTest.TestSelect:512->AuthzError:2197->
> >> AuthzError:2203->AuthzError:2220 got error:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> default.nodb
> >> expected:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> nodb.alltypes
> >>   AuthorizationTest.TestSelect:512->AuthzError:2197->
> >> AuthzError:2203->AuthzError:2220 got error:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> default.nodb
> >> expected:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> nodb.alltypes
> >>
> >>
> >>
> > Strange. In this test, we register two authorization requests, and it
> seems
> > like those are not checked in the expected order. However, that should
> not
> > be possible because we store them in a LinkedHashSet.
> > Could you dig into this a little further to see if you can figure out why
> > the order is wrong?
> >
> > This is where we register the authorization requests:
> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
> main/java/com/cloudera/impala/analysis/Analyzer.java#L544
> >
> > This is where we check the authorization requests:
> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
> main/java/com/cloudera/impala/analysis/AnalysisContext.java#L391
> >
> >
>
> I tried directly executing "select 1 from nodb.alltypes" in
> impala-shell, leading to this error:
> ERROR: AnalysisException: Could not resolv

Re: test errors in local env

[Amos Bird]

> On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird  wrote:
>
>>
>> Hi there,
>>
>> I followed the wiki
>> https://cwiki.apache.org/confluence/display/IMPALA/How+
>> to+load+and+run+Impala+tests
>> carefully but still have some problems in my local env.
>>
>> 1. I need to manually execute "hdfs dfs -mkdir /test-warehouse/emptytable"
>> to get rid of some fe test error.
>>
>>
> Ideally, you should not have to do this. Could you tell me what errors you
> encountered? Sounds like there may be a test or data loading bug we should
> fix.

The error is :

TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest)  Time elapsed: 
0.033 sec  <<< FAILURE!
java.lang.AssertionError: got error:
INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' does not 
exist.
expected:
INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' contains no 
visible files.
  at org.junit.Assert.fail(Assert.java:88)
  at org.junit.Assert.assertTrue(Assert.java:41)
  at 
com.cloudera.impala.common.FrontendTestBase.AnalysisError(FrontendTestBase.java:312)
  at 
com.cloudera.impala.common.FrontendTestBase.AnalysisError(FrontendTestBase.java:292)
  at 
com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(AnalyzeStmtsTest.java:2860)

>>
>> 2. I have authz-policy.ini in HDFS, but I still get authorization errors.
>>
>> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest)  Time
>> elapsed: 0.333 sec  <<< FAILURE!
>> java.lang.AssertionError: got error:
>> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
>> expected:
>> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
>>   at org.junit.Assert.fail(Assert.java:88)
>>   at org.junit.Assert.assertTrue(Assert.java:41)
>>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> AuthorizationTest.java:2220)
>>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> AuthorizationTest.java:2203)
>>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> AuthorizationTest.java:2197)
>>   at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
>> AuthorizationTest.java:512)
>>
>> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest)  Time
>> elapsed: 0.324 sec  <<< FAILURE!
>> java.lang.AssertionError: got error:
>> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
>> expected:
>> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
>>   at org.junit.Assert.fail(Assert.java:88)
>>   at org.junit.Assert.assertTrue(Assert.java:41)
>>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> AuthorizationTest.java:2220)
>>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> AuthorizationTest.java:2203)
>>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> AuthorizationTest.java:2197)
>>   at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
>> AuthorizationTest.java:512)
>>
>>
>> Results :
>>
>> Failed tests:
>>   AuthorizationTest.TestSelect:512->AuthzError:2197->
>> AuthzError:2203->AuthzError:2220 got error:
>> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
>> expected:
>> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
>>   AuthorizationTest.TestSelect:512->AuthzError:2197->
>> AuthzError:2203->AuthzError:2220 got error:
>> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
>> expected:
>> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
>>
>>
>>
> Strange. In this test, we register two authorization requests, and it seems
> like those are not checked in the expected order. However, that should not
> be possible because we store them in a LinkedHashSet.
> Could you dig into this a little further to see if you can figure out why
> the order is wrong?
>
> This is where we register the authorization requests:
> https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java#L544
>
> This is where we check the authorization requests:
> https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java#L391
>
>

I tried directly executing "select 1 from nodb.alltypes" in
impala-shell, leading to this error:
ERROR: AnalysisException: Could not resolve table reference: 'nodb.alltypes'

How can I reproduce the authorization tests in impala-shell so I can
debug it?


>
>>
>>
>> 3. For end-to-end tests, I encountered two kinds of errors
>>
>>   a) connection refused.
>>
>>   SET sync_ddl=False;
>> -- executing against localhost:21000
>> DROP DATABASE `test_drop_cleans_hdfs_dirs_fdfd4f8` CASCADE;
>>
>> ___ ERROR at setup of TestLoadData.test_load[exec_option:
>> {'disable_codegen': False, 'abort_on_error': 1, 
>> 'exec_single_node_rows_threshold':
>> 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none]
>> ___
>> [gw5] linux2 -- Python 2.6.6 /home/amos/i

Re: test errors in local env

[Alex Behm]

Sorry this is so difficult. We are still in the process of learning and
improving the experience for external contributors. Your help is much
appreciated.

Some responses below.

On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird  wrote:

>
> Hi there,
>
> I followed the wiki
> https://cwiki.apache.org/confluence/display/IMPALA/How+
> to+load+and+run+Impala+tests
> carefully but still have some problems in my local env.
>
> 1. I need to manually execute "hdfs dfs -mkdir /test-warehouse/emptytable"
> to get rid of some fe test error.
>
>
Ideally, you should not have to do this. Could you tell me what errors you
encountered? Sounds like there may be a test or data loading bug we should
fix.



>
> 2. I have authz-policy.ini in HDFS, but I still get authorization errors.
>
> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest)  Time
> elapsed: 0.333 sec  <<< FAILURE!
> java.lang.AssertionError: got error:
> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
> expected:
> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> AuthorizationTest.java:2220)
>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> AuthorizationTest.java:2203)
>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> AuthorizationTest.java:2197)
>   at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
> AuthorizationTest.java:512)
>
> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest)  Time
> elapsed: 0.324 sec  <<< FAILURE!
> java.lang.AssertionError: got error:
> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
> expected:
> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> AuthorizationTest.java:2220)
>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> AuthorizationTest.java:2203)
>   at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> AuthorizationTest.java:2197)
>   at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
> AuthorizationTest.java:512)
>
>
> Results :
>
> Failed tests:
>   AuthorizationTest.TestSelect:512->AuthzError:2197->
> AuthzError:2203->AuthzError:2220 got error:
> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
> expected:
> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
>   AuthorizationTest.TestSelect:512->AuthzError:2197->
> AuthzError:2203->AuthzError:2220 got error:
> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
> expected:
> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
>
>
>
Strange. In this test, we register two authorization requests, and it seems
like those are not checked in the expected order. However, that should not
be possible because we store them in a LinkedHashSet.
Could you dig into this a little further to see if you can figure out why
the order is wrong?

This is where we register the authorization requests:
https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java#L544

This is where we check the authorization requests:
https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java#L391



>
>
> 3. For end-to-end tests, I encountered two kinds of errors
>
>   a) connection refused.
>
>   SET sync_ddl=False;
> -- executing against localhost:21000
> DROP DATABASE `test_drop_cleans_hdfs_dirs_fdfd4f8` CASCADE;
>
> ___ ERROR at setup of TestLoadData.test_load[exec_option:
> {'disable_codegen': False, 'abort_on_error': 1, 
> 'exec_single_node_rows_threshold':
> 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none]
> ___
> [gw5] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
> bin/../infra/python/env/bin/python
> metadata/test_load.py:77: in setup_method
> "{0}/{1}/100101.txt".format(STAGING_PATH, i))
> util/hdfs_util.py:122: in copy
> data = self.read_file(src)
> ../infra/python/env/lib/python2.6/site-packages/pywebhdfs/webhdfs.py:183:
> in read_file
> response = requests.get(uri, allow_redirects=True)
> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:69: in get
> return request('get', url, params=params, **kwargs)
> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:50: in
> request
> response = session.request(method=method, url=url, **kwargs)
> ../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:465:
> in request
> resp = self.send(prep, **send_kwargs)
> ../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:594:
> in send
> history = [resp for resp in