[jira] [Updated] (HBASE-17429) HBase bulkload cannot support HDFS viewFs

2017-01-09 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17429:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
   2.0.0
   Status: Resolved  (was: Patch Available)

> HBase bulkload cannot support HDFS viewFs
> -
>
> Key: HBASE-17429
> URL: https://issues.apache.org/jira/browse/HBASE-17429
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.6.1, 1.2.4
> Environment: CDH5.7.0 hbase0.98.6
>Reporter: shenxianqiang
>Assignee: shenxianqiang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17429.patch
>
>
> Since hadoop cluster suppor fedration,hbase bulkload performance degrades 
> dramatically. Even if hbase directory and bulkload directory in the same 
> nameservice.
> {quote}
> 2017-01-04 21:58:40,919 INFO 
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem: use bulkload file 
> name is : hdfs://CloudTestNameNode2:8020
> 2017-01-04 21:58:40,924 ERROR org.apache.hadoop.ipc.RpcServer: Unexpected 
> throwable object 
> java.lang.IllegalArgumentException: Wrong FS: 
> viewfs://nsX/user/test/I/_tmp/9cde5dde60374b1483b7d09b65258304.top, expected: 
> hdfs://CloudTestNameNode2:8020
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:373)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:496)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:708)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3658)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3564)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionServer.java:3378)
> at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29589)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:745)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17429) HBase bulkload cannot support HDFS viewFs

2017-01-06 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17429:
---
Status: Patch Available  (was: Open)

> HBase bulkload cannot support HDFS viewFs
> -
>
> Key: HBASE-17429
> URL: https://issues.apache.org/jira/browse/HBASE-17429
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.4, 0.98.6.1
> Environment: CDH5.7.0 hbase0.98.6
>Reporter: shenxianqiang
> Attachments: HBASE-17429.patch
>
>
> Since hadoop cluster suppor fedration,hbase bulkload performance degrades 
> dramatically. Even if hbase directory and bulkload directory in the same 
> nameservice.
> {quote}
> 2017-01-04 21:58:40,919 INFO 
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem: use bulkload file 
> name is : hdfs://CloudTestNameNode2:8020
> 2017-01-04 21:58:40,924 ERROR org.apache.hadoop.ipc.RpcServer: Unexpected 
> throwable object 
> java.lang.IllegalArgumentException: Wrong FS: 
> viewfs://nsX/user/test/I/_tmp/9cde5dde60374b1483b7d09b65258304.top, expected: 
> hdfs://CloudTestNameNode2:8020
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:373)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:496)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:708)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3658)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3564)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionServer.java:3378)
> at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29589)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:745)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17429) HBase bulkload cannot support HDFS viewFs

2017-01-05 Thread shenxianqiang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

shenxianqiang updated HBASE-17429:
--
Attachment: HBASE-17429.patch

> HBase bulkload cannot support HDFS viewFs
> -
>
> Key: HBASE-17429
> URL: https://issues.apache.org/jira/browse/HBASE-17429
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.6.1, 1.2.4
> Environment: CDH5.7.0 hbase0.98.6
>Reporter: shenxianqiang
> Attachments: HBASE-17429.patch
>
>
> Since hadoop cluster suppor fedration,hbase bulkload performance degrades 
> dramatically. Even if hbase directory and bulkload directory in the same 
> nameservice.
> {quote}
> 2017-01-04 21:58:40,919 INFO 
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem: use bulkload file 
> name is : hdfs://CloudTestNameNode2:8020
> 2017-01-04 21:58:40,924 ERROR org.apache.hadoop.ipc.RpcServer: Unexpected 
> throwable object 
> java.lang.IllegalArgumentException: Wrong FS: 
> viewfs://nsX/user/test/I/_tmp/9cde5dde60374b1483b7d09b65258304.top, expected: 
> hdfs://CloudTestNameNode2:8020
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at 
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:373)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:496)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:708)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3658)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3564)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionServer.java:3378)
> at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29589)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:745)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)