[
https://issues.apache.org/jira/browse/IMPALA-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Apple resolved IMPALA-5699.
-------------------------------
Resolution: Fixed
> HBase data loading failure in Ubuntu 16.04
> ------------------------------------------
>
> Key: IMPALA-5699
> URL: https://issues.apache.org/jira/browse/IMPALA-5699
> Project: IMPALA
> Issue Type: Sub-task
> Components: Infrastructure
> Affects Versions: Impala 2.9.0
> Environment: ec2, default Ubuntu 16.04 image, r4.2xlarge, 250GB EBS
> root volue
> Reporter: Jim Apple
> Priority: Major
>
> Create an ec2 r4.2xlarge with the Ubuntu 16.04 image and a 250GB EBS root
> volume and run the script below. Data loading fails in HBase
> {noformat}
> sudo apt-get update
>
>
>
> sudo apt-get --yes install ccache ninja-build git g++ gcc git \
> libsasl2-dev libssl-dev make maven python-dev python-setuptools
> postgresql \
> liblzo2-dev
> sudo apt-get --yes install openjdk-8-jdk
> export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
> echo 'export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64' >> ~/.bashrc
> time -p git clone http://gerrit.cloudera.org:8080/Impala-ASF Impala
> cd Impala
> export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/
> echo 'export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/' >> ~/.bashrc
> sudo -u postgres psql -c "CREATE ROLE hiveuser LOGIN PASSWORD 'password';"
> postgres
> sudo -u postgres psql -c "ALTER ROLE hiveuser WITH CREATEDB;" postgres
> sudo sed -i 's/local all all
> peer/local all all
> trust/g' /etc/postgresql/9.5/main/pg_hba.conf
> sudo service postgresql restart
> sudo /etc/init.d/postgresql reload
> sudo service postgresql restart
> ssh-keygen -t rsa -N '' -q -f ~/.ssh/id_rsa
> cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
> ssh-keyscan -H github.com >> ~/.ssh/known_hosts
> echo "NoHostAuthenticationForLocalhost yes" >> ~/.ssh/config
> echo "127.0.0.1 $(hostname -s) $(hostname)" | sudo tee -a /etc/hosts
> sudo sed -i 's/127.0.1.1/127.0.0.1/g' /etc/hosts
> sudo mkdir /var/lib/hadoop-hdfs
> sudo chown $(whoami) /var/lib/hadoop-hdfs/
> echo "* hard nofile 1048576" | sudo tee -a
> /etc/security/limits.conf
> echo "* soft nofile 1048576" | sudo tee -a
> /etc/security/limits.conf
> export IMPALA_HOME="$(pwd)"
> cd ~
> git clone https://github.com/cloudera/impala-lzo.git
> ln -s impala-lzo Impala-lzo
> git clone https://github.com/cloudera/hadoop-lzo.git
> cd hadoop-lzo/
> time -p ant package
> cd "$IMPALA_HOME"
> export MAX_PYTEST_FAILURES=0
> source bin/impala-config.sh
> export NUM_CONCURRENT_TESTS=$(nproc)
> time -p ./buildall.sh -noclean -format -testdata -build_shared_libs
> {noformat}
> The output ends with
> {noformat}
> SUCCESS, data generated into /home/ubuntu/Impala/testdata/target
> Executing: create-load-data.sh
> Loading Hive Builtins (logging to
> /home/ubuntu/Impala/logs/data_loading/load-hive-builtins.log)...
> OK (Took: 1 min 3 sec)
> Generating HBase data (logging to
> /home/ubuntu/Impala/logs/data_loading/create-hbase.log)...
> OK (Took: 0 min 11 sec)
> Creating /test-warehouse HDFS directory (logging to
> /home/ubuntu/Impala/logs/data_loading/create-test-warehouse-dir.log)...
> OK (Took: 0 min 2 sec)
> Derived params for create-load-data.sh:
> EXPLORATION_STRATEGY=exhaustive
> SKIP_METADATA_LOAD=0
> SKIP_SNAPSHOT_LOAD=0
> SNAPSHOT_FILE=
> CM_HOST=
> REMOTE_LOAD=
> Starting Impala cluster (logging to
> /home/ubuntu/Impala/logs/data_loading/start-impala-cluster.log)...
> FAILED (Took: 0 min 10 sec)
> '/home/ubuntu/Impala/bin/start-impala-cluster.py
> --log_dir=/home/ubuntu/Impala/logs/data_loading -s 3' failed. Tail of log:
> Log for command '/home/ubuntu/Impala/bin/start-impala-cluster.py
> --log_dir=/home/ubuntu/Impala/logs/data_loading -s 3'
> Starting State Store logging to
> /home/ubuntu/Impala/logs/data_loading/statestored.INFO
> Starting Catalog Service logging to
> /home/ubuntu/Impala/logs/data_loading/catalogd.INFO
> Error starting cluster: Unable to start catalogd. Check log or file
> permissions for more details.
> Error in /home/ubuntu/Impala/testdata/bin/create-load-data.sh at line 48:
> LOAD_DATA_ARGS=""
> {noformat}
> {{/home/ubuntu/Impala/logs/data_loading/create-hbase.log}} ends with
> {noformat}
> 17/07/23 20:50:30 ERROR client.AsyncProcess: Failed to get region location
> org.apache.hadoop.hbase.TableNotFoundException:
> functional_hbase.hbasealltypeserror
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1404)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1199)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:410)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:359)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:238)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190)
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1498)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1082)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:450)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:311)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:59)
> at
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169)
> at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
> at
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:134)
> at
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:174)
> at
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
> at org.jruby.ast.CallManyArgsNode.interpret(CallManyArgsNode.java:59)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at
> org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:111)
> at
> org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:374)
> at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:295)
> at
> org.jruby.runtime.InterpretedBlock.yieldSpecific(InterpretedBlock.java:229)
> at org.jruby.runtime.Block.yieldSpecific(Block.java:99)
> at org.jruby.ast.ZYieldNode.interpret(ZYieldNode.java:25)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
> at
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:169)
> at
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:191)
> at
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:302)
> at
> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:144)
> at
> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:153)
> at
> org.jruby.ast.FCallNoArgBlockNode.interpret(FCallNoArgBlockNode.java:32)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:134)
> at
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:174)
> at
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
> at
> org.jruby.ast.FCallManyArgsNode.interpret(FCallManyArgsNode.java:60)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
> at
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:165)
> at org.jruby.RubyClass.finvoke(RubyClass.java:573)
> at org.jruby.RubyBasicObject.send(RubyBasicObject.java:2801)
> at org.jruby.RubyKernel.send(RubyKernel.java:2117)
> at org.jruby.RubyKernel$s$send.call(RubyKernel$s$send.gen:65535)
> at
> org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:181)
> at
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
> at
> org.jruby.ast.FCallSpecialArgNode.interpret(FCallSpecialArgNode.java:45)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at
> org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:111)
> at
> org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:374)
> at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:295)
> at
> org.jruby.runtime.InterpretedBlock.yieldSpecific(InterpretedBlock.java:229)
> at org.jruby.runtime.Block.yieldSpecific(Block.java:99)
> at org.jruby.ast.ZYieldNode.interpret(ZYieldNode.java:25)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216)
> at
> org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120)
> at org.jruby.ast.RescueNode.interpret(RescueNode.java:110)
> at
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
> at
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:165)
> at
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:272)
> at
> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:80)
> at
> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:89)
> at
> org.jruby.ast.FCallSpecialArgBlockNode.interpret(FCallSpecialArgBlockNode.java:42)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216)
> at
> org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120)
> at org.jruby.ast.RescueNode.interpret(RescueNode.java:110)
> at
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:134)
> at
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:174)
> at
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
> at
> org.jruby.ast.CallSpecialArgNode.interpret(CallSpecialArgNode.java:73)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:134)
> at
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:174)
> at
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
> at
> org.jruby.ast.FCallSpecialArgNode.interpret(FCallSpecialArgNode.java:45)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:134)
> at
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:174)
> at
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
> at
> org.jruby.ast.CallSpecialArgNode.interpret(CallSpecialArgNode.java:73)
> at org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
> at
> org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
> at
> org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:134)
> at
> org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:174)
> at
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
> at
> org.jruby.ast.FCallManyArgsNode.interpret(FCallManyArgsNode.java:60)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
> at org.jruby.ast.RootNode.interpret(RootNode.java:129)
> at
> org.jruby.evaluator.ASTInterpreter.INTERPRET_ROOT(ASTInterpreter.java:119)
> at org.jruby.Ruby.runInterpreter(Ruby.java:724)
> at org.jruby.Ruby.loadFile(Ruby.java:2489)
> at org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:66)
> at org.jruby.runtime.load.LoadService.load(LoadService.java:270)
> at org.jruby.RubyKernel.loadCommon(RubyKernel.java:1105)
> at org.jruby.RubyKernel.load(RubyKernel.java:1087)
> at
> org.jruby.RubyKernel$s$0$1$load.call(RubyKernel$s$0$1$load.gen:65535)
> at
> org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:211)
> at
> org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:207)
> at
> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169)
> at
> home.ubuntu.Impala.toolchain.cdh_components.hbase_minus_1_dot_2_dot_0_minus_cdh5_dot_13_dot_0_minus_SNAPSHOT.bin.hirb.__file__(/home/ubuntu/Impala/toolchain/cdh_components/hbase-1.2.0-cdh5.13.0-SNAPSHOT//bin/hirb.rb:177)
> at
> home.ubuntu.Impala.toolchain.cdh_components.hbase_minus_1_dot_2_dot_0_minus_cdh5_dot_13_dot_0_minus_SNAPSHOT.bin.hirb.load(/home/ubuntu/Impala/toolchain/cdh_components/hbase-1.2.0-cdh5.13.0-SNAPSHOT//bin/hirb.rb)
> at org.jruby.Ruby.runScript(Ruby.java:697)
> at org.jruby.Ruby.runScript(Ruby.java:690)
> at org.jruby.Ruby.runNormally(Ruby.java:597)
> at org.jruby.Ruby.runFromMain(Ruby.java:446)
> at org.jruby.Main.doRunFromMain(Main.java:369)
> at org.jruby.Main.internalRun(Main.java:258)
> at org.jruby.Main.run(Main.java:224)
> at org.jruby.Main.run(Main.java:208)
> at org.jruby.Main.main(Main.java:188)
> ERROR: Failed 1 action: functional_hbase.hbasealltypeserror: 1 time,
> Put a cell 'value' at specified table/row/column and optionally
> timestamp coordinates. To put a cell value into table 'ns1:t1' or 't1'
> at row 'r1' under column 'c1' marked with the time 'ts1', do:
> hbase> put 'ns1:t1', 'r1', 'c1', 'value'
> hbase> put 't1', 'r1', 'c1', 'value'
> hbase> put 't1', 'r1', 'c1', 'value', ts1
> hbase> put 't1', 'r1', 'c1', 'value', {ATTRIBUTES=>{'mykey'=>'myvalue'}}
> hbase> put 't1', 'r1', 'c1', 'value', ts1,
> {ATTRIBUTES=>{'mykey'=>'myvalue'}}
> hbase> put 't1', 'r1', 'c1', 'value', ts1, {VISIBILITY=>'PRIVATE|SECRET'}
> The same commands also can be run on a table reference. Suppose you had a
> reference
> t to table 't1', the corresponding command would be:
> hbase> t.put 'r1', 'c1', 'value', ts1, {ATTRIBUTES=>{'mykey'=>'myvalue'}}
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
