[jira] [Commented] (HBASE-27877) Hbase ImportTsv doesn't take ofs:// as a FS
[
https://issues.apache.org/jira/browse/HBASE-27877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725544#comment-17725544
]
Pratyush Bhatt commented on HBASE-27877:
Yes, works after that:
{noformat}
hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
-Dhbase.fs.tmp.dir=ofs://ozone1/vol1/bucket1/hbase/bulkload
-Dimporttsv.columns=HBASE_ROW_KEY,d:c1,d:c2
-Dimporttsv.bulk.output=ofs://ozone1/vol1/bucket1/hbase/test_VerifyHBaseNoWriteBulkloadHDFSQuota/hfiles
-Dfs.defaultFS=ofs://ozone1 table_rcb9lbo3ao
ofs://ozone1/vol1/bucket1/hbase/test_VerifyHBaseNoWriteBulkloadHDFSQuota/data.tsv{noformat}
Works fine:
{noformat}
2023-05-23 14:31:05,739|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|23/05/23 14:31:05 INFO
impl.YarnClientImpl: Submitted application application_1684781789150_0051
2023-05-23 14:31:05,794|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|23/05/23 14:31:05 INFO
mapreduce.Job: The url to track the job:
https://ozn-lease16-2.ozn-lease16.root.hwx.site:8090/proxy/application_1684781789150_0051/
2023-05-23 14:31:05,795|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|23/05/23 14:31:05 INFO
mapreduce.Job: Running job: job_1684781789150_0051
2023-05-23 14:31:20,018|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|23/05/23 14:31:20 INFO
mapreduce.Job: Job job_1684781789150_0051 running in uber mode : false
2023-05-23 14:31:20,019|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|23/05/23 14:31:20 INFO
mapreduce.Job: map 0% reduce 0%
2023-05-23 14:31:32,179|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|23/05/23 14:31:32 INFO
mapreduce.Job: map 100% reduce 0%
2023-05-23 14:31:46,301|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|23/05/23 14:31:46 INFO
mapreduce.Job: map 100% reduce 100%
2023-05-23 14:31:47,327|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|23/05/23 14:31:47 INFO
mapreduce.Job: Job job_1684781789150_0051 completed successfully
2023-05-23 14:31:47,457|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|23/05/23 14:31:47 INFO
mapreduce.Job: Counters: 54
2023-05-23 14:31:47,457|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|File System Counters
2023-05-23 14:31:47,457|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|FILE: Number of bytes read=171
2023-05-23 14:31:47,458|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|FILE: Number of bytes
written=616435
2023-05-23 14:31:47,458|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|FILE: Number of read
operations=0
2023-05-23 14:31:47,458|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|FILE: Number of large read
operations=0
2023-05-23 14:31:47,458|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|FILE: Number of write
operations=0
2023-05-23 14:31:47,458|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|OFS: Number of bytes read=257
2023-05-23 14:31:47,458|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|OFS: Number of bytes
written=5617
2023-05-23 14:31:47,458|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|OFS: Number of read
operations=12
2023-05-23 14:31:47,459|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|OFS: Number of large read
operations=0
2023-05-23 14:31:47,459|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|OFS: Number of write
operations=2
2023-05-23 14:31:47,459|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|Job Counters
2023-05-23 14:31:47,459|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|Launched map tasks=1
2023-05-23 14:31:47,459|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|Launched reduce tasks=1
2023-05-23 14:31:47,459|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|Rack-local map tasks=1
2023-05-23 14:31:47,459|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|Total time spent by all maps
in occupied slots (ms)=10227
2023-05-23 14:31:47,460|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|Total time spent by all
reduces in occupied slots (ms)=11587
2023-05-23 14:31:47,460|INFO|MainThread|machine.py:203 -
run()||GUID=f2b9a0c7-f1aa-4923-ab51-f7c9d8a92d09|Total time spent by all map
tasks (ms)=10227
2023-05-23
[jira] [Commented] (HBASE-27877) Hbase ImportTsv doesn't take ofs:// as a FS
[
https://issues.apache.org/jira/browse/HBASE-27877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725524#comment-17725524
]
Wei-Chiu Chuang commented on HBASE-27877:
-
Try specify -Dfs.defaultFS=ofs://ozone1/ and see if that addresses the issue.
> Hbase ImportTsv doesn't take ofs:// as a FS
> ---
>
> Key: HBASE-27877
> URL: https://issues.apache.org/jira/browse/HBASE-27877
> Project: HBase
> Issue Type: Bug
>Reporter: Pratyush Bhatt
>Priority: Major
> Labels: ozone
>
> While running the bulkLoad command:
> {noformat}
> hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
> -Dhbase.fs.tmp.dir=ofs://ozone1/vol1/bucket1/hbase/bulkload
> -Dimporttsv.columns=HBASE_ROW_KEY,d:c1,d:c2
> -Dimporttsv.bulk.output=ofs://ozone1/vol1/bucket1/hbase/test_VerifyHBaseNoWriteBulkloadHDFSQuota/hfiles
> table_dau3f3374e
> ofs://ozone1/vol1/bucket1/hbase/test_VerifyHBaseNoWriteBulkloadHDFSQuota/data.tsv{noformat}
> Getting:
> {noformat}
> 2023-05-22 17:01:23,263|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|Exception in thread "main"
> java.lang.IllegalArgumentException: Wrong FS:
> ofs://ozone1/vol1/bucket1/hbase/bulkload/partitions_72cbb1f1-d9b6-46a4-be39-e27a427c5842,
> expected: hdfs://ns1{noformat}
> Complete trace:
> {noformat}
> server-resourcemanager-3.1.1.7.1.8.3-339.jar:/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p3.40935426/lib/hadoop/libexec/../../hadoop-yarn/.//hadoop-yarn-registry-3.1.1.7.1.8.3-339.jar
> 2023-05-22 17:01:19,925|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ZooKeeper: Client
> environment:java.library.path=/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p3.40935426/bin/../lib/hadoop/lib/native
> 2023-05-22 17:01:19,925|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
> 2023-05-22 17:01:19,925|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ZooKeeper: Client environment:java.compiler=
> 2023-05-22 17:01:19,925|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ZooKeeper: Client environment:os.name=Linux
> 2023-05-22 17:01:19,925|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ZooKeeper: Client environment:os.arch=amd64
> 2023-05-22 17:01:19,926|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ZooKeeper: Client environment:os.version=5.4.0-135-generic
> 2023-05-22 17:01:19,926|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ZooKeeper: Client environment:user.name=hrt_qa
> 2023-05-22 17:01:19,926|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ZooKeeper: Client environment:user.home=/home/hrt_qa
> 2023-05-22 17:01:19,926|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ZooKeeper: Client environment:user.dir=/hwqe/hadoopqe
> 2023-05-22 17:01:19,926|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ZooKeeper: Client environment:os.memory.free=108MB
> 2023-05-22 17:01:19,926|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ZooKeeper: Client environment:os.memory.max=228MB
> 2023-05-22 17:01:19,927|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ZooKeeper: Client environment:os.memory.total=145MB
> 2023-05-22 17:01:19,930|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ZooKeeper: Initiating client connection,
> connectString=ozn-lease16-1.ozn-lease16.root.hwx.site:2181,ozn-lease16-2.ozn-lease16.root.hwx.site:2181,ozn-lease16-3.ozn-lease16.root.hwx.site:2181
> sessionTimeout=3
> watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$16/169226355@454e763
> 2023-05-22 17:01:19,942|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> common.X509Util: Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true
> to disable client-initiated TLS renegotiation
> 2023-05-22 17:01:19,952|INFO|MainThread|machine.py:203 -
> run()||GUID=427ca366-5f0f-426c-a2e5-0c4e12bdae2d|23/05/22 17:01:19 INFO
> zookeeper.ClientCnxnSocket:
