Hi Amos, apologies for the delay? Is this issue still open? If so, can you try to run the tests with Java 7?
Thanks, Lars On Thu, Sep 22, 2016 at 6:38 PM, Amos Bird <[email protected]> wrote: > Thanks Lars, > > My Java version is > > $ $JAVA_HOME/bin/java -version > java version "1.8.0_65" > Java(TM) SE Runtime Environment (build 1.8.0_65-b17) > Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode) > > > Hi Amos, > > > > I tried to see if IMPALA-3643 > > <https://issues.cloudera.org/browse/IMPALA-3643> is still reproducible > for > > me. In the process I also ran into the "NoClassDefFoundError" error you > saw > > and found out, that this happens if my local hadoop services > > (./testdata/bin/run-all.sh) isn't running. > > > > Once I have the local hadoop services running I'm able to repro the issue > > in IMPALA-3643. The only thing I remember doing differently from the > > recommended ways of dev-machine setup is using Java 8. The Java 8 docs on > > LinkedHashSet > > <https://docs.oracle.com/javase/8/docs/api/java/util/LinkedHashSet.html> > don't > > read like they'd change the ordering, however it doesn't say that the > > ordering has to be stable *between different versions of Java*. I will > try > > to confirm that this issue does not repro with Java 7 when I have time. > Can > > you check the Java version you're running. Mine is this: > > > > $ $JAVA_HOME/bin/java -version > > java version "1.8.0_101" > > Java(TM) SE Runtime Environment (build 1.8.0_101-b13) > > Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode) > > > > Alex, could a difference in the implementation between Java 7 and 8 be a > > reasonable explanation for this? > > > > Cheers, Lars > > > > On Thu, Sep 22, 2016 at 12:52 AM, Amos Bird <[email protected]> wrote: > > > >> > >> Thank you for helping me out :D > >> > >> > Thanks for keeping this discussion going. > >> > > >> > On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird <[email protected]> > wrote: > >> > > >> >> > >> >> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird <[email protected]> > >> wrote: > >> >> > > >> >> >> > >> >> >> Hi there, > >> >> >> > >> >> >> I followed the wiki > >> >> >> https://cwiki.apache.org/confluence/display/IMPALA/How+ > >> >> >> to+load+and+run+Impala+tests > >> >> >> carefully but still have some problems in my local env. > >> >> >> > >> >> >> 1. I need to manually execute "hdfs dfs -mkdir > >> >> /test-warehouse/emptytable" > >> >> >> to get rid of some fe test error. > >> >> >> > >> >> >> > >> >> > Ideally, you should not have to do this. Could you tell me what > errors > >> >> you > >> >> > encountered? Sounds like there may be a test or data loading bug we > >> >> should > >> >> > fix. > >> >> > >> >> The error is : > >> >> > >> >> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest) Time > >> >> elapsed: 0.033 sec <<< FAILURE! > >> >> java.lang.AssertionError: got error: > >> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' > does > >> >> not exist. > >> >> expected: > >> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' > >> >> contains no visible files. > >> >> at org.junit.Assert.fail(Assert.java:88) > >> >> at org.junit.Assert.assertTrue(Assert.java:41) > >> >> at com.cloudera.impala.common.FrontendTestBase.AnalysisError( > >> >> FrontendTestBase.java:312) > >> >> at com.cloudera.impala.common.FrontendTestBase.AnalysisError( > >> >> FrontendTestBase.java:292) > >> >> at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData( > >> >> AnalyzeStmtsTest.java:2860) > >> >> > >> >> > >> > Do you have a table functional.emptytable? If yes, then what location > is > >> > reported in "show create table"? > >> Query: show create table functional.emptytable > >> +-------------------------------------------------------------+ > >> | result | > >> +-------------------------------------------------------------+ > >> | CREATE EXTERNAL TABLE functional.emptytable ( | > >> | field STRING | > >> | ) | > >> | PARTITIONED BY ( | > >> | f2 INT | > >> | ) | > >> | STORED AS TEXTFILE | > >> | LOCATION 'hdfs://localhost:20500/test-warehouse/emptytable' | > >> | TBLPROPERTIES ('transient_lastDdlTime'='1464782625') | > >> +-------------------------------------------------------------+ > >> Fetched 1 row(s) in 5.51s > >> > >> > Does the directory exist in HDFS? > >> No. > >> > >> > > >> > You could try to manually reload the table and see if the directory is > >> > created: > >> > bin/load-data.py -f -w functional-query --table_names=emptytable > >> > --table_formats=text/none > >> After executing this command the directory appears. > >> > >> > > >> >>> > >> >> >> 2. I have authz-policy.ini in HDFS, but I still get authorization > >> >> errors. > >> >> >> > >> >> >> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest) > Time > >> >> >> elapsed: 0.333 sec <<< FAILURE! > >> >> >> java.lang.AssertionError: got error: > >> >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> >> default.nodb > >> >> >> expected: > >> >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> >> nodb.alltypes > >> >> >> at org.junit.Assert.fail(Assert.java:88) > >> >> >> at org.junit.Assert.assertTrue(Assert.java:41) > >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError( > >> >> >> AuthorizationTest.java:2220) > >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError( > >> >> >> AuthorizationTest.java:2203) > >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError( > >> >> >> AuthorizationTest.java:2197) > >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect( > >> >> >> AuthorizationTest.java:512) > >> >> >> > >> >> >> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest) > Time > >> >> >> elapsed: 0.324 sec <<< FAILURE! > >> >> >> java.lang.AssertionError: got error: > >> >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> >> default.nodb > >> >> >> expected: > >> >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> >> nodb.alltypes > >> >> >> at org.junit.Assert.fail(Assert.java:88) > >> >> >> at org.junit.Assert.assertTrue(Assert.java:41) > >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError( > >> >> >> AuthorizationTest.java:2220) > >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError( > >> >> >> AuthorizationTest.java:2203) > >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError( > >> >> >> AuthorizationTest.java:2197) > >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect( > >> >> >> AuthorizationTest.java:512) > >> >> >> > >> >> >> > >> >> >> Results : > >> >> >> > >> >> >> Failed tests: > >> >> >> AuthorizationTest.TestSelect:512->AuthzError:2197-> > >> >> >> AuthzError:2203->AuthzError:2220 got error: > >> >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> >> default.nodb > >> >> >> expected: > >> >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> >> nodb.alltypes > >> >> >> AuthorizationTest.TestSelect:512->AuthzError:2197-> > >> >> >> AuthzError:2203->AuthzError:2220 got error: > >> >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> >> default.nodb > >> >> >> expected: > >> >> >> User 'amos' does not have privileges to execute 'SELECT' on: > >> >> nodb.alltypes > >> >> >> > >> >> >> > >> >> >> > >> >> > Strange. In this test, we register two authorization requests, and > it > >> >> seems > >> >> > like those are not checked in the expected order. However, that > should > >> >> not > >> >> > be possible because we store them in a LinkedHashSet. > >> >> > Could you dig into this a little further to see if you can figure > out > >> why > >> >> > the order is wrong? > >> >> > > >> >> > This is where we register the authorization requests: > >> >> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/ > >> >> main/java/com/cloudera/impala/analysis/Analyzer.java#L544 > >> >> > > >> >> > This is where we check the authorization requests: > >> >> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/ > >> >> main/java/com/cloudera/impala/analysis/AnalysisContext.java#L391 > >> >> > > >> >> > > >> >> > >> >> I tried directly executing "select 1 from nodb.alltypes" in > >> >> impala-shell, leading to this error: > >> >> ERROR: AnalysisException: Could not resolve table reference: > >> >> 'nodb.alltypes' > >> >> > >> >> How can I reproduce the authorization tests in impala-shell so I can > >> >> debug it? > >> >> > >> >> > >> >> > >> > FYI, this is actually a known issue and may have something to do with > the > >> > JRE version you are running: https://issues.cloudera.org/ > >> browse/IMPALA-3643 > >> > As far as I can tell the bug should be "impossible" because we use a > >> > LinkedHashSet, but maybe certain JREs do not properly honor the > >> guarantees. > >> > > >> > The AuthorizationTests in particular require a non-trivial setup, so > I'd > >> > not recommend trying to debug via the Impala shell. > >> > > >> > I'd recommend debugging in one of these ways: > >> > - Run the test manually via "mvn test -Dtest=AuthorizationTest" from > the > >> FE > >> > directory. Attach debugger and break in TestSelect(). > >> > - Run the JUnit test from an IDE such as Eclipse and then debug the > test. > >> > I'm afraid there is no easy way to just run that single query in our > >> > current test setup. You will need to run the whole suite, but you can > >> break > >> > TestSelect() or hack the code in various places to set useful > >> breakpoints. > >> > > >> > Hope that helps. > >> > >> I tried jdk1.8.0_102. Compilation works just fine but AuthorizationTest > >> fails with ClassNoDefine. > >> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.008 > sec > >> <<< FAILURE! - in com.cloudera.impala.analysis.AuthorizationTest > >> initializationError(com.cloudera.impala.analysis.AuthorizationTest) > Time > >> elapsed: 0.006 sec <<< ERROR! > >> java.lang.NoClassDefFoundError: Could not initialize class > >> com.cloudera.impala.analysis.AuthorizationTest > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at sun.reflect.NativeMethodAccessorImpl.invoke( > >> NativeMethodAccessorImpl.java:62) > >> at sun.reflect.DelegatingMethodAccessorImpl.invoke( > >> DelegatingMethodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:498) > >> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( > >> FrameworkMethod.java:50) > >> at org.junit.internal.runners.model.ReflectiveCallable.run( > >> ReflectiveCallable.java:12) > >> at org.junit.runners.model.FrameworkMethod.invokeExplosively( > >> FrameworkMethod.java:47) > >> at org.junit.runners.Parameterized.allParameters( > Parameterized.java:280) > >> at org.junit.runners.Parameterized.<init>(Parameterized.java:248) > >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > >> at sun.reflect.NativeConstructorAccessorImpl.newInstance( > >> NativeConstructorAccessorImpl.java:62) > >> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > >> DelegatingConstructorAccessorImpl.java:45) > >> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > >> at org.junit.internal.builders.AnnotatedBuilder.buildRunner( > >> AnnotatedBuilder.java:104) > >> at org.junit.internal.builders.AnnotatedBuilder.runnerForClass( > >> AnnotatedBuilder.java:86) > >> at org.junit.runners.model.RunnerBuilder.safeRunnerForClass( > >> RunnerBuilder.java:59) > >> at org.junit.internal.builders.AllDefaultPossibilitiesBuilder > >> .runnerForClass(AllDefaultPossibilitiesBuilder.java:26) > >> at org.junit.runners.model.RunnerBuilder.safeRunnerForClass( > >> RunnerBuilder.java:59) > >> at org.junit.internal.requests.ClassRequest.getRunner( > >> ClassRequest.java:33) > >> at org.apache.maven.surefire.junit4.JUnit4Provider.execute( > >> JUnit4Provider.java:283) > >> at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun( > >> JUnit4Provider.java:173) > >> at org.apache.maven.surefire.junit4.JUnit4Provider. > >> executeTestSet(JUnit4Provider.java:153) > >> at org.apache.maven.surefire.junit4.JUnit4Provider.invoke( > >> JUnit4Provider.java:128) > >> at org.apache.maven.surefire.booter.ForkedBooter. > >> invokeProviderInSameClassLoader(ForkedBooter.java:203) > >> at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess( > >> ForkedBooter.java:155) > >> at org.apache.maven.surefire.booter.ForkedBooter.main( > >> ForkedBooter.java:103) > >> > >> > > >> >> > >> >> >> > >> >> >> > >> >> >> 3. For end-to-end tests, I encountered two kinds of errors > >> >> >> > >> >> >> a) connection refused. > >> >> >> > >> >> >> SET sync_ddl=False; > >> >> >> -- executing against localhost:21000 > >> >> >> DROP DATABASE `test_drop_cleans_hdfs_dirs_fdfd4f8` CASCADE; > >> >> >> > >> >> >> ___________________ ERROR at setup of TestLoadData.test_load[exec_ > >> >> option: > >> >> >> {'disable_codegen': False, 'abort_on_error': 1, > >> 'exec_single_node_rows_ > >> >> threshold': > >> >> >> 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] > >> >> >> ___________________ > >> >> >> [gw5] linux2 -- Python 2.6.6 /home/amos/incubator-impala/ > >> >> >> bin/../infra/python/env/bin/python > >> >> >> metadata/test_load.py:77: in setup_method > >> >> >> "{0}/{1}/100101.txt".format(STAGING_PATH, i)) > >> >> >> util/hdfs_util.py:122: in copy > >> >> >> data = self.read_file(src) > >> >> >> ../infra/python/env/lib/python2.6/site-packages/ > >> >> pywebhdfs/webhdfs.py:183: > >> >> >> in read_file > >> >> >> response = requests.get(uri, allow_redirects=True) > >> >> >> ../infra/python/env/lib/python2.6/site-packages/ > requests/api.py:69: > >> in > >> >> get > >> >> >> return request('get', url, params=params, **kwargs) > >> >> >> ../infra/python/env/lib/python2.6/site-packages/ > requests/api.py:50: > >> in > >> >> >> request > >> >> >> response = session.request(method=method, url=url, **kwargs) > >> >> >> ../infra/python/env/lib/python2.6/site-packages/ > >> >> requests/sessions.py:465: > >> >> >> in request > >> >> >> resp = self.send(prep, **send_kwargs) > >> >> >> ../infra/python/env/lib/python2.6/site-packages/ > >> >> requests/sessions.py:594: > >> >> >> in send > >> >> >> history = [resp for resp in gen] if allow_redirects else [] > >> >> >> ../infra/python/env/lib/python2.6/site-packages/ > >> >> requests/sessions.py:196: > >> >> >> in resolve_redirects > >> >> >> **adapter_kwargs > >> >> >> ../infra/python/env/lib/python2.6/site-packages/ > >> >> requests/sessions.py:573: > >> >> >> in send > >> >> >> r = adapter.send(request, **kwargs) > >> >> >> ../infra/python/env/lib/python2.6/site-packages/ > >> >> requests/adapters.py:415: > >> >> >> in send > >> >> >> raise ConnectionError(err, request=request) > >> >> >> E ConnectionError: ('Connection aborted.', error(111, > 'Connection > >> >> >> refused')) > >> >> >> > >> >> >> > >> >> > The connection refused issue is very bizarre. One thing that I > >> noticed is > >> >> > that your Python does not seem to match what we use (Python 2.7.3). > >> >> > Could you re-run infra/python/bootstrap_virtualenv.py and see if > you > >> get > >> >> > the expected version into infra/python/env/local/bin? > >> >> > > >> >> > Alternatively, maybe there's a problem with your /etc/hosts? You > can > >> try > >> >> > searching online for WebHdfs and /etc/hosts > >> >> > > >> >> > >> >> well, I find this 'find_py26.py' file under deps. Is it normal? > >> >> > >> > > >> > Yes, that's normal. That file looks Python 2.6 on your system but > should > >> > not be relevant for running tests because we use the Python from our > >> > virtualenv and not the one on your system. > >> > > >> > What's your output when you run "impala-python --version". You should > get > >> > 'Python 2.7.3". > >> > >> [amos@t450s tests]$ impala-python --version > >> Python 2.6.6 > >> > >> > Also, what's the Python version on your system? Our virtualenv will > use > >> > Python 2.6 if your system has a Python < 2.6. > >> > You could try to upgrade your system Python and then > >> > re-run infra/python/bootstrap_virtualenv.py > >> > >> My system has python 2.6.6. > >> > >> > > >> > Still, theoretically Python 2.6 in the virtual env should work. I > think > >> > it's more likely you are having a connection problem due to a > >> misconfigured > >> > /etc/hosts. > >> > > >> > Are you running the test from a shell that has bin/impala-config.sh > and > >> > bin/set-classpath.sh sourced? > >> Yes. > >> > >> > > >> > To further debug this you could try to specify your namenode address > when > >> > running the test to see whether it is somehow picking up a wrong > address: > >> > cd tests > >> > ./run-tests.py metadata/test_load.py --namenode_http_address= > >> localhost:50070 > >> > > >> > And see if that works. > >> > >> Unfortunately, no. > >> > >> > > >> > > >> > > >> >> [amos@nobida143 incubator-impala]$ ls infra/python/deps/ > >> >> download_requirements find_py26.py pip_download.py > requirements.txt > >> >> [amos@nobida143 incubator-impala]$ cat infra/python/deps/download_ > >> >> requirements > >> >> #!/bin/bash > >> >> > >> >> # Licensed to the Apache Software Foundation (ASF) under one > >> >> # or more contributor license agreements. See the NOTICE file > >> >> # distributed with this work for additional information > >> >> # regarding copyright ownership. The ASF licenses this file > >> >> # to you under the Apache License, Version 2.0 (the > >> >> # "License"); you may not use this file except in compliance > >> >> # with the License. You may obtain a copy of the License at > >> >> # > >> >> # http://www.apache.org/licenses/LICENSE-2.0 > >> >> # > >> >> # Unless required by applicable law or agreed to in writing, > >> >> # software distributed under the License is distributed on an > >> >> # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY > >> >> # KIND, either express or implied. See the License for the > >> >> # specific language governing permissions and limitations > >> >> # under the License. > >> >> > >> >> set -euo pipefail > >> >> > >> >> DIR="$(dirname "$0")" > >> >> > >> >> pushd "$DIR" > >> >> PY26="$(./find_py26.py)" > >> >> # Directly download packages listed in requirements.txt, but don't > >> install > >> >> them. > >> >> "$PY26" pip_download.py > >> >> # For virtualenv, other scripts rely on the .tar.gz package (not a > .whl > >> >> package). > >> >> "$PY26" pip_download.py virtualenv 13.1.0 > >> >> # kudu-python is downloaded separately because pip install attempts > to > >> >> execute a > >> >> # setup.py subcommand for kudu-python that can fail even if the > download > >> >> succeeds. > >> >> "$PY26" pip_download.py kudu-python 0.1.1 > >> >> popd > >> >> > >> >> > >> >> > >> >> > b) stats not match > >> >> >> > >> >> >> [gw4] linux2 -- Python 2.6.6 /home/amos/incubator-impala/ > >> >> >> bin/../infra/python/env/bin/python > >> >> >> metadata/test_metadata_query_statements.py:67: in test_show_stats > >> >> >> self.run_test_case('QueryTest/show-stats', vector, > "functional") > >> >> >> common/impala_test_suite.py:342: in run_test_case > >> >> >> self.__verify_results_and_errors(vector, test_section, > result, > >> >> use_db) > >> >> >> common/impala_test_suite.py:234: in __verify_results_and_errors > >> >> >> replace_filenames_with_placeholder) > >> >> >> common/test_result_verifier.py:398: in verify_raw_results > >> >> >> VERIFIER_MAP[verifier](expected, actual) > >> >> >> common/test_result_verifier.py:231: in > verify_query_result_is_equal > >> >> >> assert expected_results == actual_results > >> >> >> > >> >> >> ... > >> >> >> > >> >> >> -- executing against localhost:21000 > >> >> >> show column stats alltypes_clone; > >> >> >> > >> >> >> MainThread: Comparing QueryTestResults (expected vs actual): > >> >> >> 'bigint_col','BIGINT',10,-1,8,8 == 'bigint_col','BIGINT',10,-1,8, > 8 > >> >> >> 'bool_col','BOOLEAN',2,-1,1,1 == 'bool_col','BOOLEAN',2,-1,1,1 > >> >> >> 'date_string_col','STRING',736,-1,8,8 == > 'date_string_col','STRING', > >> >> >> 736,-1,8,8 > >> >> >> 'double_col','DOUBLE',-1,-1,8,8 == 'double_col','DOUBLE',-1,-1,8, > 8 > >> >> >> 'float_col','FLOAT',10,-1,4,4 == 'float_col','FLOAT',10,-1,4,4 > >> >> >> 'id','INT',7505,-1,4,4 == 'id','INT',7505,-1,4,4 > >> >> >> 'int_col','INT',-1,-1,4,4 == 'int_col','INT',-1,-1,4,4 > >> >> >> 'month','INT',12,0,4,4 == 'month','INT',12,0,4,4 > >> >> >> 'smallint_col','SMALLINT',10,-1,2,2 == > >> 'smallint_col','SMALLINT',10,- > >> >> 1,2,2 > >> >> >> 'string_col','STRING',10,-1,-1,-1 == > 'string_col','STRING',10,-1,- > >> 1,-1 > >> >> >> 'timestamp_col','TIMESTAMP',7554,-1,16,16 != > >> >> 'timestamp_col','TIMESTAMP', > >> >> >> 7552,-1,16,16 > >> >> >> 'tinyint_col','TINYINT',10,-1,1,1 == > 'tinyint_col','TINYINT',10,-1, > >> 1,1 > >> >> >> 'year','INT',2,0,4,4 == 'year','INT',2,0,4,4 > >> >> >> > >> >> >> > >> >> >> Very strange. Can you do a compute stats on functional.alltypes > and > >> >> > confirm that the NDV for timestamp_col are 7552 in your setup? > >> >> > >> >> Yes. > >> >> > >> > > >> > I'll need to ask around for help. I have no idea why this is > happening. > >> > >> Thanks :) > >> > >> > > >> >> > >> >> > > >> >> > > >> >> > > >> >> >> I'm using CentOS 6.8 final. I have no idea what goes wrong. Any > help > >> is > >> >> >> much appreciated! > >> >> > > >> >> > > >> >> > > >> >> > > >> >> >> > >> >> >> Best regards, > >> >> >> Amos > >> >> >> > >> >> > >> >> > >> > >> > >
