[ 
https://issues.apache.org/jira/browse/IMPALA-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jim Apple resolved IMPALA-5699.
-------------------------------
    Resolution: Fixed

> HBase data loading failure in Ubuntu 16.04
> ------------------------------------------
>
>                 Key: IMPALA-5699
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5699
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Infrastructure
>    Affects Versions: Impala 2.9.0
>         Environment: ec2, default Ubuntu 16.04 image, r4.2xlarge, 250GB EBS 
> root volue
>            Reporter: Jim Apple
>            Priority: Major
>
> Create an ec2 r4.2xlarge with the Ubuntu 16.04 image and a 250GB EBS root 
> volume and run the script below. Data loading fails in HBase 
> {noformat}
> sudo apt-get update                                                           
>                                                                               
>                                                                               
>     
> sudo apt-get --yes install ccache ninja-build git g++ gcc git \  
>      libsasl2-dev libssl-dev make maven python-dev python-setuptools 
> postgresql \  
>      liblzo2-dev
> sudo apt-get --yes install openjdk-8-jdk
> export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
> echo 'export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64' >> ~/.bashrc
> time -p git clone http://gerrit.cloudera.org:8080/Impala-ASF Impala
> cd Impala
> export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/
> echo 'export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/' >> ~/.bashrc
> sudo -u postgres psql -c "CREATE ROLE hiveuser LOGIN PASSWORD 'password';" 
> postgres
> sudo -u postgres psql -c "ALTER ROLE hiveuser WITH CREATEDB;" postgres
> sudo sed -i 's/local   all             all                                    
>  peer/local   all             all                                     
> trust/g' /etc/postgresql/9.5/main/pg_hba.conf
> sudo service postgresql restart
> sudo /etc/init.d/postgresql reload
> sudo service postgresql restart
> ssh-keygen -t rsa -N '' -q -f ~/.ssh/id_rsa
> cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
> ssh-keyscan -H github.com >> ~/.ssh/known_hosts
> echo "NoHostAuthenticationForLocalhost yes" >> ~/.ssh/config
> echo "127.0.0.1 $(hostname -s) $(hostname)" | sudo tee -a /etc/hosts
> sudo sed -i 's/127.0.1.1/127.0.0.1/g' /etc/hosts
> sudo mkdir /var/lib/hadoop-hdfs
> sudo chown $(whoami) /var/lib/hadoop-hdfs/
> echo "*               hard    nofile          1048576" | sudo tee -a 
> /etc/security/limits.conf
> echo "*               soft    nofile          1048576" | sudo tee -a 
> /etc/security/limits.conf
> export IMPALA_HOME="$(pwd)"
> cd ~
> git clone https://github.com/cloudera/impala-lzo.git
> ln -s impala-lzo Impala-lzo
> git clone https://github.com/cloudera/hadoop-lzo.git
> cd hadoop-lzo/
> time -p ant package
> cd "$IMPALA_HOME"
> export MAX_PYTEST_FAILURES=0
> source bin/impala-config.sh
> export NUM_CONCURRENT_TESTS=$(nproc)
> time -p ./buildall.sh -noclean -format -testdata -build_shared_libs
> {noformat}
> The output ends with
> {noformat}
> SUCCESS, data generated into /home/ubuntu/Impala/testdata/target
> Executing: create-load-data.sh 
> Loading Hive Builtins (logging to 
> /home/ubuntu/Impala/logs/data_loading/load-hive-builtins.log)... 
>     OK (Took: 1 min 3 sec)
> Generating HBase data (logging to 
> /home/ubuntu/Impala/logs/data_loading/create-hbase.log)... 
>     OK (Took: 0 min 11 sec)
> Creating /test-warehouse HDFS directory (logging to 
> /home/ubuntu/Impala/logs/data_loading/create-test-warehouse-dir.log)... 
>     OK (Took: 0 min 2 sec)
> Derived params for create-load-data.sh:
> EXPLORATION_STRATEGY=exhaustive
> SKIP_METADATA_LOAD=0
> SKIP_SNAPSHOT_LOAD=0
> SNAPSHOT_FILE=
> CM_HOST=
> REMOTE_LOAD=
> Starting Impala cluster (logging to 
> /home/ubuntu/Impala/logs/data_loading/start-impala-cluster.log)... 
>     FAILED (Took: 0 min 10 sec)
>     '/home/ubuntu/Impala/bin/start-impala-cluster.py 
> --log_dir=/home/ubuntu/Impala/logs/data_loading -s 3' failed. Tail of log:
> Log for command '/home/ubuntu/Impala/bin/start-impala-cluster.py 
> --log_dir=/home/ubuntu/Impala/logs/data_loading -s 3'
> Starting State Store logging to 
> /home/ubuntu/Impala/logs/data_loading/statestored.INFO
> Starting Catalog Service logging to 
> /home/ubuntu/Impala/logs/data_loading/catalogd.INFO
> Error starting cluster: Unable to start catalogd. Check log or file 
> permissions for more details.
> Error in /home/ubuntu/Impala/testdata/bin/create-load-data.sh at line 48: 
> LOAD_DATA_ARGS=""
> {noformat}
> {{/home/ubuntu/Impala/logs/data_loading/create-hbase.log}} ends with
> {noformat}
> 17/07/23 20:50:30 ERROR client.AsyncProcess: Failed to get region location 
> org.apache.hadoop.hbase.TableNotFoundException: 
> functional_hbase.hbasealltypeserror
>         at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1404)
>         at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1199)
>         at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:410)
>         at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:359)
>         at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:238)
>         at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190)
>         at 
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1498)
>         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1082)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at 
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:450)
>         at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:311)
>         at 
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:59)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169)
>         at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
>         at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
>         at 
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:134)
>         at 
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:174)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
>         at org.jruby.ast.CallManyArgsNode.interpret(CallManyArgsNode.java:59)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
>         at 
> org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:111)
>         at 
> org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:374)
>         at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:295)
>         at 
> org.jruby.runtime.InterpretedBlock.yieldSpecific(InterpretedBlock.java:229)
>         at org.jruby.runtime.Block.yieldSpecific(Block.java:99)
>         at org.jruby.ast.ZYieldNode.interpret(ZYieldNode.java:25)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
>         at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
>         at 
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:169)
>         at 
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:191)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:302)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:144)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:153)
>         at 
> org.jruby.ast.FCallNoArgBlockNode.interpret(FCallNoArgBlockNode.java:32)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
>         at 
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:134)
>         at 
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:174)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
>         at 
> org.jruby.ast.FCallManyArgsNode.interpret(FCallManyArgsNode.java:60)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
>         at 
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
>         at 
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:165)
>         at org.jruby.RubyClass.finvoke(RubyClass.java:573)
>         at org.jruby.RubyBasicObject.send(RubyBasicObject.java:2801)
>         at org.jruby.RubyKernel.send(RubyKernel.java:2117)
>         at org.jruby.RubyKernel$s$send.call(RubyKernel$s$send.gen:65535)
>         at 
> org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:181)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
>         at 
> org.jruby.ast.FCallSpecialArgNode.interpret(FCallSpecialArgNode.java:45)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
>         at 
> org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:111)
>         at 
> org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:374)
>         at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:295)
>         at 
> org.jruby.runtime.InterpretedBlock.yieldSpecific(InterpretedBlock.java:229)
>         at org.jruby.runtime.Block.yieldSpecific(Block.java:99)
>         at org.jruby.ast.ZYieldNode.interpret(ZYieldNode.java:25)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
>         at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216)
>         at 
> org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120)
>         at org.jruby.ast.RescueNode.interpret(RescueNode.java:110)
>         at 
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
>         at 
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:165)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:272)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:80)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:89)
>         at 
> org.jruby.ast.FCallSpecialArgBlockNode.interpret(FCallSpecialArgBlockNode.java:42)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
>         at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216)
>         at 
> org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120)
>         at org.jruby.ast.RescueNode.interpret(RescueNode.java:110)
>         at 
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:134)
>         at 
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:174)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
>         at 
> org.jruby.ast.CallSpecialArgNode.interpret(CallSpecialArgNode.java:73)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
>         at 
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:134)
>         at 
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:174)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
>         at 
> org.jruby.ast.FCallSpecialArgNode.interpret(FCallSpecialArgNode.java:45)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
>         at 
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:134)
>         at 
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:174)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
>         at 
> org.jruby.ast.CallSpecialArgNode.interpret(CallSpecialArgNode.java:73)
>         at org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
>         at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
>         at 
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
>         at 
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:134)
>         at 
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:174)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
>         at 
> org.jruby.ast.FCallManyArgsNode.interpret(FCallManyArgsNode.java:60)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
>         at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
>         at org.jruby.ast.RootNode.interpret(RootNode.java:129)
>         at 
> org.jruby.evaluator.ASTInterpreter.INTERPRET_ROOT(ASTInterpreter.java:119)
>         at org.jruby.Ruby.runInterpreter(Ruby.java:724)
>         at org.jruby.Ruby.loadFile(Ruby.java:2489)
>         at org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:66)
>         at org.jruby.runtime.load.LoadService.load(LoadService.java:270)
>         at org.jruby.RubyKernel.loadCommon(RubyKernel.java:1105)
>         at org.jruby.RubyKernel.load(RubyKernel.java:1087)
>         at 
> org.jruby.RubyKernel$s$0$1$load.call(RubyKernel$s$0$1$load.gen:65535)
>         at 
> org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:211)
>         at 
> org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:207)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312)
>         at 
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169)
>         at 
> home.ubuntu.Impala.toolchain.cdh_components.hbase_minus_1_dot_2_dot_0_minus_cdh5_dot_13_dot_0_minus_SNAPSHOT.bin.hirb.__file__(/home/ubuntu/Impala/toolchain/cdh_components/hbase-1.2.0-cdh5.13.0-SNAPSHOT//bin/hirb.rb:177)
>         at 
> home.ubuntu.Impala.toolchain.cdh_components.hbase_minus_1_dot_2_dot_0_minus_cdh5_dot_13_dot_0_minus_SNAPSHOT.bin.hirb.load(/home/ubuntu/Impala/toolchain/cdh_components/hbase-1.2.0-cdh5.13.0-SNAPSHOT//bin/hirb.rb)
>         at org.jruby.Ruby.runScript(Ruby.java:697)
>         at org.jruby.Ruby.runScript(Ruby.java:690)
>         at org.jruby.Ruby.runNormally(Ruby.java:597)
>         at org.jruby.Ruby.runFromMain(Ruby.java:446)
>         at org.jruby.Main.doRunFromMain(Main.java:369)
>         at org.jruby.Main.internalRun(Main.java:258)
>         at org.jruby.Main.run(Main.java:224)
>         at org.jruby.Main.run(Main.java:208)
>         at org.jruby.Main.main(Main.java:188)
> ERROR: Failed 1 action: functional_hbase.hbasealltypeserror: 1 time, 
> Put a cell 'value' at specified table/row/column and optionally
> timestamp coordinates.  To put a cell value into table 'ns1:t1' or 't1'
> at row 'r1' under column 'c1' marked with the time 'ts1', do:
>   hbase> put 'ns1:t1', 'r1', 'c1', 'value'
>   hbase> put 't1', 'r1', 'c1', 'value'
>   hbase> put 't1', 'r1', 'c1', 'value', ts1
>   hbase> put 't1', 'r1', 'c1', 'value', {ATTRIBUTES=>{'mykey'=>'myvalue'}}
>   hbase> put 't1', 'r1', 'c1', 'value', ts1, 
> {ATTRIBUTES=>{'mykey'=>'myvalue'}}
>   hbase> put 't1', 'r1', 'c1', 'value', ts1, {VISIBILITY=>'PRIVATE|SECRET'}
> The same commands also can be run on a table reference. Suppose you had a 
> reference
> t to table 't1', the corresponding command would be:
>   hbase> t.put 'r1', 'c1', 'value', ts1, {ATTRIBUTES=>{'mykey'=>'myvalue'}}
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to