taklwu commented on pull request #3634:
URL: https://github.com/apache/hbase/pull/3634#issuecomment-906777428
this commit is required for the completion of HBASE-25853, otherwise the
import TSV nightly job with shaded client artifact fail with
NoClassDefFoundError exception. and I tested the nightly tests locally with
the following steps (with outputs)
```
# first maven full build
HBASE_SOURCE=/Users/taklwu/workspace/hbase2
cd $HBASE_SOURCE
mvn clean install -DskipTests -Prelease && mvn install -DskipTests site
assembly:single -Prelease
# unzip the assembly and hbase-client
cd ~/hbase-nightly
install_artifact=$(ls -1
${HBASE_SOURCE}/hbase-assembly/target/hbase-*-bin.tar.gz | sort | head -n 1)
mkdir hbase-install
tar --strip-component=1 -xzf "${install_artifact}" -C "hbase-install"
client_artifact=$(ls -1
${HBASE_SOURCE}/hbase-assembly/target/hbase-*-bin.tar.gz | sort | tail -n 1)
mkdir hbase-client
tar --strip-component=1 -xzf "${client_artifact}" -C "hbase-client"
# run the nightly
rm -rf output-integration/hadoop-2/*
bash $HBASE_SOURCE/dev-support/hbase_nightly_pseudo-distributed-test.sh
--single-process --working-dir output-integration/hadoop-2
--hbase-client-install hbase-client hbase-install hadoop-2/bin/hadoop
hadoop-2/share/hadoop/yarn/timelineservice
hadoop-2/share/hadoop/yarn/test/hadoop-yarn-server-tests-2.10.1-tests.jar
hadoop-2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.1-tests.jar
hadoop-2/bin/mapred | tee ~/hbase-nightly/sanity-check.log
Using HBase client-side artifact
You'll find logs and temp files in
/Users/taklwu/hbase-nightly/output-integration/hadoop-2
Hadoop version information:
Hadoop 2.10.1
Subversion https://github.com/apache/hadoop -r
1827467c9a56f133025f28557bfc2c562d78e816
Compiled by centos on 2020-09-14T13:17Z
Compiled with protoc 2.5.0
From source with checksum 3114edef868f1f3824e7d0f68be03650
This command was run using
/Users/taklwu/hbase-nightly/hadoop-2/share/hadoop/common/hadoop-common-2.10.1.jar
JAVA_HOME: /Users/taklwu/.sdkman/candidates/java/8.292.10.1-amzn
HBase version information:
HBase 2.5.0-SNAPSHOT
Source code repository git://21992.local/Users/taklwu/workspace/hbase2
revision=723831a9de938faf7c2a1b2e4047b68ed281af52
Compiled by taklwu on Thu Aug 26 14:29:11 PDT 2021
From source with checksum
c90bbbd1a0293de7e3621195fad2d2e6ade1a2b82fdcb05ee7690e3fb8b338a743d7428dee3ef18b9c6b6d7de1ea979bc8073f5b9bb686b24fc48aa4e8b48e90
Writing out configuration for HBase.
Starting up Hadoop
waiting for Hadoop to finish starting up.
waiting for Hadoop to finish starting up.
waiting for Hadoop to finish starting up.
Listing HDFS contents
Starting up HBase
running master, logging to
/Users/taklwu/hbase-nightly/hbase-install/bin/../logs/hbase-taklwu-master-21992.local.out
retry waiting for hbase to come up.
Setting up table 'test:example' with 1,000 regions
writing out example TSV to example.tsv
uploading example.tsv to HDFS
2021-08-26 14:59:41,249 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2021-08-26 14:59:42,671 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Importing TSV via shaded client artifact for HBase - MapReduce integration.
Verifying row count from import.
Hadoop client jars not given; getting them from 'hadoop classpath' for the
example.
Building shaded client example.
Running shaded client example. It'll fetch the set of regions, round-trip
them to a file in HDFS, then write them one-per-row into the test table.
Checking on results of example program.
2021-08-26 15:02:29,669 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Verifying row count from example.
Found 1051 rows, which is enough to cover 48 for import, 1000 example's use
of user table regions, 2 for example's use of meta/namespace regions, and 1 for
example's count record
Shutting down HBase
stopping hbase............
Shutdown: listing HDFS contents
Shutting down Hadoop
# check the command exit code and output of mr-importtsv.err
% echo $?
0
% cat output-integration/hadoop-2/mr-importtsv.err
...
2021-08-26 15:00:00,482 INFO [main] mapreduce.Job: Job
job_local361623923_0001 completed successfully
2021-08-26 15:00:00,487 INFO [main] mapreduce.Job: Counters: 21
File System Counters
FILE: Number of bytes read=47342109
FILE: Number of bytes written=48290866
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=756
HDFS: Number of bytes written=0
HDFS: Number of read operations=3
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Map-Reduce Framework
Map input records=48
Map output records=48
Input split bytes=119
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=0
Total committed heap usage (bytes)=556793856
ImportTsv
Bad Lines=0
File Input Format Counters
Bytes Read=756
File Output Format Counters
Bytes Written=0
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
