[jira] [Resolved] (HBASE-15203) Reduce garbage created by path.toString() during Checksum verification
[ https://issues.apache.org/jira/browse/HBASE-15203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan resolved HBASE-15203. Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.3.0 Resolving this. I don't think this is needed in branch-1.1 because Path is not passed for checksum calculation. > Reduce garbage created by path.toString() during Checksum verification > -- > > Key: HBASE-15203 > URL: https://issues.apache.org/jira/browse/HBASE-15203 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15203.patch, HBASE-15203_1.patch, > HBASE-15203_2.patch, HBASE-15203_branch-1.1.patch > > > When we try to read a block we do checksum verification for which we need the > file name in which the block belongs to. So we do Path.toString() every time. > This seems to create around 163MB of char[] that is garbage collected in a > simple scan run. It is also visible in writes but the impact is lesser. In > overall write/read profile the top 2 factors are byte[] and char[]. This > toString() can easily be avoided and reduce its share from the total. To make > it more precise in 1 min of profiling, among the 1.8G of garbage created by > StringBuilder.toString - this path.toString() was contributing around 3.5%. > After the patch this is totally not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15252) Data loss when replaying wal if HDFS timeout
[
https://issues.apache.org/jira/browse/HBASE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-15252:
Fix Version/s: (was: 1.2.1)
1.2.0
> Data loss when replaying wal if HDFS timeout
>
>
> Key: HBASE-15252
> URL: https://issues.apache.org/jira/browse/HBASE-15252
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15252-testcase.patch, HBASE-15252-v1.patch,
> HBASE-15252.patch
>
>
> This is a problem introduced by HBASE-13825 where we change the exception
> type in catch block in {{readNext}} method of {{ProtobufLogReader}}.
> {code:title=ProtobufLogReader.java}
> try {
> ..
> ProtobufUtil.mergeFrom(builder, new
> LimitInputStream(this.inputStream, size),
> (int)size);
> } catch (IOException ipbe) { // <-- used to be
> InvalidProtocolBufferException
> throw (EOFException) new EOFException("Invalid PB, EOF? Ignoring;
> originalPosition=" +
> originalPosition + ", currentPosition=" +
> this.inputStream.getPos() +
> ", messageSize=" + size + ", currentAvailable=" +
> available).initCause(ipbe);
> }
> {code}
> Here if the {{inputStream}} throws an {{IOException}} due to timeout or
> something, we just convert it to an {{EOFException}} and at the bottom of
> this method, we ignore {{EOFException}} and return false. This cause the
> upper layer think we reach the end of file. So when replaying we will treat
> the HDFS timeout error as a normal end of file and cause data loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15203) Reduce garbage created by path.toString() during Checksum verification
[ https://issues.apache.org/jira/browse/HBASE-15203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144115#comment-15144115 ] Nick Dimiduk commented on HBASE-15203: -- Thanks Ram. > Reduce garbage created by path.toString() during Checksum verification > -- > > Key: HBASE-15203 > URL: https://issues.apache.org/jira/browse/HBASE-15203 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15203.patch, HBASE-15203_1.patch, > HBASE-15203_2.patch, HBASE-15203_branch-1.1.patch > > > When we try to read a block we do checksum verification for which we need the > file name in which the block belongs to. So we do Path.toString() every time. > This seems to create around 163MB of char[] that is garbage collected in a > simple scan run. It is also visible in writes but the impact is lesser. In > overall write/read profile the top 2 factors are byte[] and char[]. This > toString() can easily be avoided and reduce its share from the total. To make > it more precise in 1 min of profiling, among the 1.8G of garbage created by > StringBuilder.toString - this path.toString() was contributing around 3.5%. > After the patch this is totally not there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14046) IllegalArgumentException: Negative position from hfile.bucket.FileIOEngine.read
[
https://issues.apache.org/jira/browse/HBASE-14046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-14046:
-
Component/s: (was: hbase)
regionserver
> IllegalArgumentException: Negative position from
> hfile.bucket.FileIOEngine.read
> ---
>
> Key: HBASE-14046
> URL: https://issues.apache.org/jira/browse/HBASE-14046
> Project: HBase
> Issue Type: Bug
> Components: BucketCache, regionserver
>Affects Versions: 0.98.6
> Environment: CentOS release 6.4 X86_64
>Reporter: louis hust
>Priority: Critical
>
> When I scan a table using hbase shell, got the following message:
> {code:bash}
> hbase(main):001:0> scan 'atpco:ttf_record6'
> ROW COLUMN+CELL
> ERROR: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> Expected nextCallSeq: 1 But the nextCallSeq got from client: 0;
> request=scanner_id: 201542113 number_of_rows: 100 close_scanner: false
> next_call_seq: 0
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3193)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:744)
> {code}
> And the region server got the following error:
> {code}
> 2015-06-30 11:08:11,877 ERROR
> [B.defaultRpcServer.handler=27,queue=0,port=60020] ipc.RpcServer: Unexpected
> throwable object
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:675)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.read(FileIOEngine.java:87)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:406)
> at
> org.apache.hadoop.hbase.io.hfile.LruBlockCache.getBlock(LruBlockCache.java:389)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:635)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:749)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:136)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:507)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:140)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3900)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3980)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3858)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3849)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3245)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> {code}
> It seems something wrong with the BucketCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144102#comment-15144102
]
Hudson commented on HBASE-15198:
FAILURE: Integrated in HBase-1.2 #545 (See
[https://builds.apache.org/job/HBase-1.2/545/])
HBASE-15198 RPC client not using Codec and CellBlock for puts by (anoopsamjohn:
rev 19d964d637ab808fad5ea011419b1b07bad53665)
*
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch,
> HBASE-15198_addendum-branch-1.0.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-14046) IllegalArgumentException: Negative position from hfile.bucket.FileIOEngine.read
[
https://issues.apache.org/jira/browse/HBASE-14046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan reassigned HBASE-14046:
--
Assignee: ramkrishna.s.vasudevan
> IllegalArgumentException: Negative position from
> hfile.bucket.FileIOEngine.read
> ---
>
> Key: HBASE-14046
> URL: https://issues.apache.org/jira/browse/HBASE-14046
> Project: HBase
> Issue Type: Bug
> Components: BucketCache, regionserver
>Affects Versions: 0.98.6
> Environment: CentOS release 6.4 X86_64
>Reporter: louis hust
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
>
> When I scan a table using hbase shell, got the following message:
> {code:bash}
> hbase(main):001:0> scan 'atpco:ttf_record6'
> ROW COLUMN+CELL
> ERROR: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> Expected nextCallSeq: 1 But the nextCallSeq got from client: 0;
> request=scanner_id: 201542113 number_of_rows: 100 close_scanner: false
> next_call_seq: 0
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3193)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> at java.lang.Thread.run(Thread.java:744)
> {code}
> And the region server got the following error:
> {code}
> 2015-06-30 11:08:11,877 ERROR
> [B.defaultRpcServer.handler=27,queue=0,port=60020] ipc.RpcServer: Unexpected
> throwable object
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:675)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.read(FileIOEngine.java:87)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:406)
> at
> org.apache.hadoop.hbase.io.hfile.LruBlockCache.getBlock(LruBlockCache.java:389)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:635)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:749)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:136)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:507)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:140)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3900)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3980)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3858)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3849)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3245)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> {code}
> It seems something wrong with the BucketCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15167) Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
[
https://issues.apache.org/jira/browse/HBASE-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144116#comment-15144116
]
Nick Dimiduk commented on HBASE-15167:
--
Thanks [~chenheng].
> Deadlock in TestNamespaceAuditor.testRegionOperations on 1.1
>
>
> Key: HBASE-15167
> URL: https://issues.apache.org/jira/browse/HBASE-15167
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 1.1.3
>Reporter: Nick Dimiduk
>Assignee: Heng Chen
>Priority: Critical
> Fix For: 1.1.4
>
> Attachments: HBASE-15167-branch-1.1.patch, blocked.log
>
>
> This was left as a zombie after one of my test runs this weekend.
> {noformat}
> "WALProcedureStoreSyncThread" daemon prio=10 tid=0x7f3ccc209000
> nid=0x3960 in Object.wait() [0x7f3c6b6b5000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> - locked <0x0007f2813390> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy23.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:264)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy24.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1612)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1488)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1413)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:387)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:383)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:327)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:766)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:733)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.tryRollWriter(WALProcedureStore.java:668)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.periodicRoll(WALProcedureStore.java:711)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:531)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:66)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:180)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15248) One block, one seek: a.k.a BLOCKSIZE 4k should result in 4096 bytes on disk
[ https://issues.apache.org/jira/browse/HBASE-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144130#comment-15144130 ] Nick Dimiduk commented on HBASE-15248: -- To further confuse, I believe our alignment while serializing a block is calculated before compression is applied, not after. So if you have BLOCKSIZE=64k and COMPRESSION=GZ, you end up with something a lot smaller. > One block, one seek: a.k.a BLOCKSIZE 4k should result in 4096 bytes on disk > --- > > Key: HBASE-15248 > URL: https://issues.apache.org/jira/browse/HBASE-15248 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > Chatting w/ a gentleman named Daniel Pol who is messing w/ bucketcache, he > wants blocks to be the size specified in the configuration and no bigger. His > hardware set ups fetches pages of 4k and so a block that has 4k of payload > but has then a header and the header of the next block (which helps figure > whats next when scanning) ends up being 4203 bytes or something, and this > then then translates into two seeks per block fetch. > This issue is about what it would take to stay inside our configured size > boundary writing out blocks. > If not possible, give back better signal on what to do so you could fit > inside a particular constraint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144140#comment-15144140
]
Hudson commented on HBASE-15198:
SUCCESS: Integrated in HBase-1.3 #551 (See
[https://builds.apache.org/job/HBase-1.3/551/])
HBASE-15198 RPC client not using Codec and CellBlock for puts by (anoopsamjohn:
rev 7e5ce6b0a3b893cbb7282f54ac11f10775e80fc3)
*
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch,
> HBASE-15198_addendum-branch-1.0.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15140) Fix ResultBoundedCompletionService deadlock
[
https://issues.apache.org/jira/browse/HBASE-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-15140:
-
Resolution: Duplicate
Status: Resolved (was: Patch Available)
Pushed cherry-pick of HBASE-14812 to branch-1.1. Closing this one as duplicate.
Thanks for pointing it out [~alok_at_cloudability].
> Fix ResultBoundedCompletionService deadlock
>
>
> Key: HBASE-15140
> URL: https://issues.apache.org/jira/browse/HBASE-15140
> Project: HBase
> Issue Type: Bug
> Components: Client, hbase
>Affects Versions: 1.1.2
>Reporter: Alok Singh
>Priority: Critical
> Fix For: 1.1.4
>
> Attachments: 14812-cherrypick-branch-1.1.patch
>
>
> See: https://issues.apache.org/jira/browse/HBASE-14812
> We ran into this deadlock issue on the hbase 1.1.2 build
> {code}
> phoenix-1-thread-1340 id=3183 state=WAITING
> - waiting on <0x1ad981d3> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x1ad981d3> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at
> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:88)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:79)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Locked synchronizers: count = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@55145434
> phoenix-1-thread-1341 id=3184 state=WAITING
> - waiting on <0x22e46b2c> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x22e46b2c> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at
> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:88)
> at
> org.apache.phoenix.iterate.TableResultIterator.(TableResultIterator.java:79)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at
> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
>
[jira] [Updated] (HBASE-14812) Fix ResultBoundedCompletionService deadlock
[
https://issues.apache.org/jira/browse/HBASE-14812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-14812:
-
Fix Version/s: 1.1.4
Apply also to 1.1.
> Fix ResultBoundedCompletionService deadlock
> ---
>
> Key: HBASE-14812
> URL: https://issues.apache.org/jira/browse/HBASE-14812
> Project: HBase
> Issue Type: Bug
> Components: Client, Thrift
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>
> Attachments: HBASE-14812-v1.patch, HBASE-14812-v2.patch,
> HBASE-14812.patch
>
>
> {code}
> "thrift2-worker-0" #31 daemon prio=5 os_prio=0 tid=0x7f5ad9c45000
> nid=0x484 in Object.wait() [0x7f5aa3832000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> - locked <0x0006b6ae7670> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
> at
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1276)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1182)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:194)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:171)
> - locked <0x0006b6ae79c0> (a
> org.apache.hadoop.hbase.client.BufferedMutatorImpl)
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1033)
> at
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler.putMultiple(ThriftHBaseServiceHandler.java:268)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler$THBaseServiceMetricsProxy.invoke(ThriftHBaseServiceHandler.java:114)
> at com.sun.proxy.$Proxy10.putMultiple(Unknown Source)
> at
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1637)
> at
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1621)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> at
> org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15120) Backport HBASE-14883 to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-15120: - Attachment: HBASE-15120.branch-1.1.patch Once more, for HadoopQA. > Backport HBASE-14883 to branch-1.1 > -- > > Key: HBASE-15120 > URL: https://issues.apache.org/jira/browse/HBASE-15120 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Fix For: 1.1.4 > > Attachments: HBASE-15120.branch-1.1.patch, > HBASE-15120.branch-1.1.patch, HBASE-15120.branch-1.1.patch > > > When checking branch-1.1 UT in HBASE-13590, found > TestSplitTransactionOnCluster#testFailedSplit will fail at a 12/50 chance. > The issue is fixed by HBASE-14883 but the change didn't go into branch-1.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15120) Backport HBASE-14883 to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-15120: - Fix Version/s: 1.1.4 > Backport HBASE-14883 to branch-1.1 > -- > > Key: HBASE-15120 > URL: https://issues.apache.org/jira/browse/HBASE-15120 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Fix For: 1.1.4 > > Attachments: HBASE-15120.branch-1.1.patch, > HBASE-15120.branch-1.1.patch, HBASE-15120.branch-1.1.patch > > > When checking branch-1.1 UT in HBASE-13590, found > TestSplitTransactionOnCluster#testFailedSplit will fail at a 12/50 chance. > The issue is fixed by HBASE-14883 but the change didn't go into branch-1.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144173#comment-15144173
]
Hudson commented on HBASE-15198:
SUCCESS: Integrated in HBase-1.3-IT #497 (See
[https://builds.apache.org/job/HBase-1.3-IT/497/])
HBASE-15198 RPC client not using Codec and CellBlock for puts by (anoopsamjohn:
rev 7e5ce6b0a3b893cbb7282f54ac11f10775e80fc3)
*
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch,
> HBASE-15198_addendum-branch-1.0.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15169) Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-15169: - Attachment: HBASE-15169-branch-1.1.patch Still not run. Again for build bot. > Backport HBASE-14362 'TestWALProcedureStoreOnHDFS is super duper flaky' to > branch-1.1 > - > > Key: HBASE-15169 > URL: https://issues.apache.org/jira/browse/HBASE-15169 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Nick Dimiduk >Assignee: Heng Chen >Priority: Critical > Fix For: 1.1.4 > > Attachments: HBASE-15169-branch-1.1.patch, > HBASE-15169-branch-1.1.patch, HBASE-15169-branch-1.1.patch > > > This one fails consistently for me and there's a fix hanging out upstream. > Let's bring it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144186#comment-15144186
]
Nick Dimiduk commented on HBASE-15198:
--
Nice find folks.
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch,
> HBASE-15198_addendum-branch-1.0.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15259) WALEdits under replay will also be replicated
ramkrishna.s.vasudevan created HBASE-15259:
--
Summary: WALEdits under replay will also be replicated
Key: HBASE-15259
URL: https://issues.apache.org/jira/browse/HBASE-15259
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Priority: Minor
I need to verify this but seeing the code
{code}
try {
// We are about to append this edit; update the region-scoped sequence
number. Do it
// here inside this single appending/writing thread. Events are
ordered on the ringbuffer
// so region sequenceids will also be in order.
regionSequenceId = entry.stampRegionSequenceId();
// Edits are empty, there is nothing to append. Maybe empty when we
are looking for a
// region sequence id only, a region edit/sequence id that is not
associated with an actual
// edit. It has to go through all the rigmarole to be sure we have the
right ordering.
if (entry.getEdit().isEmpty()) {
return;
}
// Coprocessor hook.
if (!coprocessorHost.preWALWrite(entry.getHRegionInfo(), entry.getKey(),
entry.getEdit())) {
if (entry.getEdit().isReplay()) {
// Set replication scope null so that this won't be replicated
entry.getKey().setScopes(null);
}
}
if (!listeners.isEmpty()) {
for (WALActionsListener i: listeners) {
// TODO: Why does listener take a table description and CPs take a
regioninfo? Fix.
i.visitLogEntryBeforeWrite(entry.getHTableDescriptor(),
entry.getKey(),
entry.getEdit());
}
}
{code}
When a WALEdit is in replay we set the Logkey to null. But in the
visitLogEntryBeforeWrite() we again set the LogKey based on the replication
scope associated with the cells. So the previous step of setting null does not
work here?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)
[ https://issues.apache.org/jira/browse/HBASE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144209#comment-15144209 ] Hudson commented on HBASE-15221: SUCCESS: Integrated in HBase-1.1-JDK8 #1748 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1748/]) HBASE-15221 HTableMultiplexer improvements (stale region locations and (tedyu: rev 360a5782c19bd61df9e6b92cc2a72ff0260a082d) * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexerViaMocks.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java * hbase-client/pom.xml > HTableMultiplexer improvements (stale region locations and resource leaks) > -- > > Key: HBASE-15221 > URL: https://issues.apache.org/jira/browse/HBASE-15221 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4 > > Attachments: HBASE-15221.001.patch, HBASE-15221.002.patch, > HBASE-15221.003.patch, HBASE-15221.branch-1.patch, HBASE-15221.v4.patch > > > It looks like HTableMultiplexer has a couple of issues. > Upon failing to send a Put to the appropriate RS, the Put is re-queued back > into the system. Normally this is fine as such an exception is transient and > the Put would eventually succeed. However, in the case where the Put was > rejected because of a NotServingRegionException (e.g. split, balance, merge), > the re-queuing of the Put will end up using the same cached HRegionLocation. > This means that the Put will just be repeatedly sent back to the same RS over > and over again, eventually being dropped on the floor. Need to invalidate the > location cache (or make sure we refresh it) when we re-queue the Put. > The internal ClusterConnection is also leaked. If a user creates many > HTableMultiplexers, they'll eventually run into issues (memory, zk > connections, etc) because they'll never get cleaned up. HTableMultiplexer > needs a close method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15251) During a cluster restart, Hmaster thinks it is a failover by mistake
[
https://issues.apache.org/jira/browse/HBASE-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144215#comment-15144215
]
Clara Xiong commented on HBASE-15251:
-
Thank you for the feedback. I will update line 465,
For line 447, I removed the proceeding line "} else {" so failover is true
after finding dead server. This is a further check when dead servers are found.
So I flip it until I actually see any regions on these dead servers.
For line 575, I do expect directory for path dir so it is a warning. Same for
line 585.
Any comments for any user case I might be missing? I thought about these two
checks and believe states passing these two checks should go to the clean start
logic.
> During a cluster restart, Hmaster thinks it is a failover by mistake
>
>
> Key: HBASE-15251
> URL: https://issues.apache.org/jira/browse/HBASE-15251
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 2.0.0, 0.98.15
>Reporter: Clara Xiong
>Assignee: Clara Xiong
> Attachments: HBASE-15251-master.patch
>
>
> We often need to do cluster restart as part of release for a cluster of >
> 1000 nodes. We have tried our best to get clean shutdown but 50% of the time,
> hmaster still thinks it is a failover. This increases the restart time from 5
> min to 30 min and decreases locality from 99% to 5% since we didn't use a
> locality-aware balancer. We had a bug HBASE-14129 but the fix didn't work.
> After adding more logging and inspecting the logs, we identified two things
> that trigger the failover handling:
> 1. When Hmaster.AssignmentManager detects any dead servers on service
> manager during joinCluster(), it determines this is a failover without
> further check. I added a check whether there is even any region assigned to
> these servers. During a clean restart, the regions are not even assigned.
> 2. When there are some leftover empty folders for log and split directories
> or empty wal files, it is also treated as a failover. I added a check for
> that. Although this can be resolved by manual cleanup, it is still too
> tedious for restarting a large cluster.
> Patch will follow shortly. The fix is tested and used in production now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14490) [RpcServer] reuse request read buffer
[
https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144218#comment-15144218
]
Anoop Sam John commented on HBASE-14490:
bq.Do we need to create another pool for read and write seperately? IMHO I feel
yes.
Ya read reqs into one pool's buffer and write into another's... that is not
really possible. Because we don't know the op type here in RPC. May be we can
have one large buffer pool and one small buffer pool. Write reqs normally land
in the large category. The worry was reserving more memory as we have 2 pools.
bq.Moreover, as you know, when you use a heap buffer the Oracle implementation
uses an internal direct buffer pool and copies data between them, and not only
that might cause shortage of the off-heap area, but also that causes overhead
of redundant copying.
Yes. That is why along with HBASE-11425 work, we changed the BBBPool to be off
heap rather than on heap. Did some perf testing also that time. Having the
BBBPool making on heap buffers and caching having some -ve impact in G1GC
noticed by Elliot. I forgot which Jira he says it.
bq.I prefer to use a pool of direct buffers whose have a fixed length,
NIO_BUFFER_LIMIT, that is assumed to be larger than the size of the buffer of a
native socket.
When u say this, u mean when we have a write req with size larger than this
buffer's size, read it into N buffers rather than one and use? I had some
thinking in that direction and had some offline discussion with Stack and Ram.
Also was working on a PoC patch there. We are actively working in this area
to see what best suits us. Will be able to tell more in near future.
bq.So here the getBuffer will create a direct buffer if the queue is empty and
that DBB we will throw away if the capacity is not enough? Such offheap memory
area when will it get cleared ?
Ya that is another concern for me. Off heap buffers we make and those are not
pooled... If we end up in many such allocations and throw away, the full GC,
when needed will have an -ve impact? When we are not able to get a buffer from
our BBBPool (these buffers are off heap), we should be creating only on heap
buffers? data = ByteBuffer.allocate(dataLength); -> Here I can see it is on
heap. I mean when we ask Pool to give one and it is not able to find (it's
capacity is reached and buffer is not available). It is becoming more imp in
request path as we know the size need.
> [RpcServer] reuse request read buffer
> -
>
> Key: HBASE-14490
> URL: https://issues.apache.org/jira/browse/HBASE-14490
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 2.0.0, 1.0.2
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
> Labels: performance
> Fix For: 2.0.0, 1.0.2
>
> Attachments: 14490.hack.to.1.2.patch, ByteBufferPool.java,
> HBASE-14490-v1.patch, HBASE-14490-v10.patch, HBASE-14490-v11.patch,
> HBASE-14490-v12.patch, HBASE-14490-v2.patch, HBASE-14490-v3.patch,
> HBASE-14490-v4.patch, HBASE-14490-v5.patch, HBASE-14490-v6.patch,
> HBASE-14490-v7.patch, HBASE-14490-v8.patch, HBASE-14490-v9.patch, gc.png,
> hits.png, test-v12-patch
>
>
> Reusing buffer to read request.It's not necessary to every request free
> buffer.The idea of optimization is to reduce the times that allocate
> ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}},
> {{processConnectionHeader}} accept a ByteBuffer instead of byte[].They can
> move {{ByteBuffer.position}} correctly when we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a buffer pool in each {{Connection}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14490) [RpcServer] reuse request read buffer
[
https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144219#comment-15144219
]
Hiroshi Ikeda commented on HBASE-14490:
---
{quote}
but I was rejected in some other JIRA issue if I remember right.
Where Hiroshi Ikeda We want to save on the offheap copy to oneheap from socket
for sure.
{quote}
What I said above is for writing (HBASE-14873). BoundedByteBuffer is just used
for writing data to a socket, as far as I know, and GatheringByteChannel can
accept multiple byte buffers so that discarding and re-creating direct buffers,
which might cause fragmentation of the off-heap area, is not necessary.
As to reading, in general, data for an operation has a various length, which
might be quite smaller or larger than NIO_BUFFER_LIMIT, and it doesn't seem
effective to hold a given direct buffer to prepare and execute its operation.
Besides the issue of copying, using another pool for large heap buffers might
be useful to reduce GC frequency, as described long ago.
> [RpcServer] reuse request read buffer
> -
>
> Key: HBASE-14490
> URL: https://issues.apache.org/jira/browse/HBASE-14490
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 2.0.0, 1.0.2
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
> Labels: performance
> Fix For: 2.0.0, 1.0.2
>
> Attachments: 14490.hack.to.1.2.patch, ByteBufferPool.java,
> HBASE-14490-v1.patch, HBASE-14490-v10.patch, HBASE-14490-v11.patch,
> HBASE-14490-v12.patch, HBASE-14490-v2.patch, HBASE-14490-v3.patch,
> HBASE-14490-v4.patch, HBASE-14490-v5.patch, HBASE-14490-v6.patch,
> HBASE-14490-v7.patch, HBASE-14490-v8.patch, HBASE-14490-v9.patch, gc.png,
> hits.png, test-v12-patch
>
>
> Reusing buffer to read request.It's not necessary to every request free
> buffer.The idea of optimization is to reduce the times that allocate
> ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}},
> {{processConnectionHeader}} accept a ByteBuffer instead of byte[].They can
> move {{ByteBuffer.position}} correctly when we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a buffer pool in each {{Connection}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-15198:
---
Fix Version/s: (was: 1.2.1)
1.2.0
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch,
> HBASE-15198_addendum-branch-1.0.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144221#comment-15144221
]
Anoop Sam John commented on HBASE-15198:
Changed fix version from 1.2.1 to 1.2.0 as we will be having new RC. Already
pushed to branch-1.2..JFYI [~busbey]
Issue is open for push to 0.98.
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch,
> HBASE-15198_addendum-branch-1.0.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14490) [RpcServer] reuse request read buffer
[
https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144225#comment-15144225
]
Anoop Sam John commented on HBASE-14490:
bq.What I said above is for writing (HBASE-14873). BoundedByteBuffer is just
used for writing data to a socket, as far as I know, and GatheringByteChannel
can accept multiple byte buffers so that discarding and re-creating direct
buffers, which might cause fragmentation of the off-heap area, is not necessary.
Ya while writing it is not really necessary to have the entire CellBlock in one
BB. It can be N buffers. We can avoid the need for recreate and copy as the
size of the block becomes more.. Did some work in this area already.. I need
to make the code better and do some testing.Was thinking for reading (I
mean request read) how/whether we can use this strategy so that we can have
fixed sized buffer pool..
> [RpcServer] reuse request read buffer
> -
>
> Key: HBASE-14490
> URL: https://issues.apache.org/jira/browse/HBASE-14490
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 2.0.0, 1.0.2
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
> Labels: performance
> Fix For: 2.0.0, 1.0.2
>
> Attachments: 14490.hack.to.1.2.patch, ByteBufferPool.java,
> HBASE-14490-v1.patch, HBASE-14490-v10.patch, HBASE-14490-v11.patch,
> HBASE-14490-v12.patch, HBASE-14490-v2.patch, HBASE-14490-v3.patch,
> HBASE-14490-v4.patch, HBASE-14490-v5.patch, HBASE-14490-v6.patch,
> HBASE-14490-v7.patch, HBASE-14490-v8.patch, HBASE-14490-v9.patch, gc.png,
> hits.png, test-v12-patch
>
>
> Reusing buffer to read request.It's not necessary to every request free
> buffer.The idea of optimization is to reduce the times that allocate
> ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}},
> {{processConnectionHeader}} accept a ByteBuffer instead of byte[].They can
> move {{ByteBuffer.position}} correctly when we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a buffer pool in each {{Connection}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14490) [RpcServer] reuse request read buffer
[
https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143897#comment-15143897
]
Anoop Sam John edited comment on HBASE-14490 at 2/12/16 7:53 AM:
-
{code}
} finally {
1612dataLengthBuffer.clear(); // Clean for the next call
1613reqBufPool.putBuffer(data);
1614data = null;
1615 }
{code}
We can not put back the buffer to pool here. The req is just given to handlers
for processing. With CellScanners in place, we are not sure whether all Cells
are read out of it yet. We can release only at completeion of the Call. This
was initially proposed for read reqs and now for all read and write.
We need 2 pools for req and response?
FYI Stack, we are working in this area for the write path off heaping.. Will
be doing some perf testing soon.
Correction:
We can release the buffers where the req is read into after
resultPair = this.rpcServer.call(call.service, call.md, call.param,
call.cellScanner,
call.timestamp, this.status);
We have done with the req and we are about to make the response...
was (Author: anoop.hbase):
{code}
} finally {
1612dataLengthBuffer.clear(); // Clean for the next call
1613reqBufPool.putBuffer(data);
1614data = null;
1615 }
{code}
We can not put back the buffer to pool here. The req is just given to handlers
for processing. With CellScanners in place, we are not sure whether all Cells
are read out of it yet. We can release only at completeion of the Call. This
was initially proposed for read reqs and now for all read and write.
We need 2 pools for req and response?
FYI Stack, we are working in this area for the write path off heaping.. Will
be doing some perf testing soon.
> [RpcServer] reuse request read buffer
> -
>
> Key: HBASE-14490
> URL: https://issues.apache.org/jira/browse/HBASE-14490
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 2.0.0, 1.0.2
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
> Labels: performance
> Fix For: 2.0.0, 1.0.2
>
> Attachments: 14490.hack.to.1.2.patch, ByteBufferPool.java,
> HBASE-14490-v1.patch, HBASE-14490-v10.patch, HBASE-14490-v11.patch,
> HBASE-14490-v12.patch, HBASE-14490-v2.patch, HBASE-14490-v3.patch,
> HBASE-14490-v4.patch, HBASE-14490-v5.patch, HBASE-14490-v6.patch,
> HBASE-14490-v7.patch, HBASE-14490-v8.patch, HBASE-14490-v9.patch, gc.png,
> hits.png, test-v12-patch
>
>
> Reusing buffer to read request.It's not necessary to every request free
> buffer.The idea of optimization is to reduce the times that allocate
> ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}},
> {{processConnectionHeader}} accept a ByteBuffer instead of byte[].They can
> move {{ByteBuffer.position}} correctly when we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a buffer pool in each {{Connection}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14490) [RpcServer] reuse request read buffer
[
https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144068#comment-15144068
]
ramkrishna.s.vasudevan commented on HBASE-14490:
{code}
data = reqBufPool.getBuffer();
1574if (data.capacity() < dataLength) {
1575 data = ByteBuffer.allocate(dataLength);
1576} else {
1577 data.limit(dataLength);
1578}
{code}
So here the getBuffer will create a direct buffer if the queue is empty and
that DBB we will throw away if the capacity is not enough? Such offheap memory
area when will it get cleared ?
> [RpcServer] reuse request read buffer
> -
>
> Key: HBASE-14490
> URL: https://issues.apache.org/jira/browse/HBASE-14490
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 2.0.0, 1.0.2
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
> Labels: performance
> Fix For: 2.0.0, 1.0.2
>
> Attachments: 14490.hack.to.1.2.patch, ByteBufferPool.java,
> HBASE-14490-v1.patch, HBASE-14490-v10.patch, HBASE-14490-v11.patch,
> HBASE-14490-v12.patch, HBASE-14490-v2.patch, HBASE-14490-v3.patch,
> HBASE-14490-v4.patch, HBASE-14490-v5.patch, HBASE-14490-v6.patch,
> HBASE-14490-v7.patch, HBASE-14490-v8.patch, HBASE-14490-v9.patch, gc.png,
> hits.png, test-v12-patch
>
>
> Reusing buffer to read request.It's not necessary to every request free
> buffer.The idea of optimization is to reduce the times that allocate
> ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}},
> {{processConnectionHeader}} accept a ByteBuffer instead of byte[].They can
> move {{ByteBuffer.position}} correctly when we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a buffer pool in each {{Connection}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15252) Data loss when replaying wal if HDFS timeout
[
https://issues.apache.org/jira/browse/HBASE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-15252:
Priority: Blocker (was: Major)
> Data loss when replaying wal if HDFS timeout
>
>
> Key: HBASE-15252
> URL: https://issues.apache.org/jira/browse/HBASE-15252
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15252-testcase.patch, HBASE-15252-v1.patch,
> HBASE-15252.patch
>
>
> This is a problem introduced by HBASE-13825 where we change the exception
> type in catch block in {{readNext}} method of {{ProtobufLogReader}}.
> {code:title=ProtobufLogReader.java}
> try {
> ..
> ProtobufUtil.mergeFrom(builder, new
> LimitInputStream(this.inputStream, size),
> (int)size);
> } catch (IOException ipbe) { // <-- used to be
> InvalidProtocolBufferException
> throw (EOFException) new EOFException("Invalid PB, EOF? Ignoring;
> originalPosition=" +
> originalPosition + ", currentPosition=" +
> this.inputStream.getPos() +
> ", messageSize=" + size + ", currentAvailable=" +
> available).initCause(ipbe);
> }
> {code}
> Here if the {{inputStream}} throws an {{IOException}} due to timeout or
> something, we just convert it to an {{EOFException}} and at the bottom of
> this method, we ignore {{EOFException}} and return false. This cause the
> upper layer think we reach the end of file. So when replaying we will treat
> the HDFS timeout error as a normal end of file and cause data loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-15198:
---
Attachment: HBASE-15198-0.98.patch
HBASE-15198-branch-1.0.patch
HBASE-15198-branch-1.1.patch
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-branch-1.0.patch,
> HBASE-15198-branch-1.1.patch, HBASE-15198-branch-1.patch, HBASE-15198.patch,
> HBASE-15198_V2.patch, HBASE-15198_V3.patch, HBASE-15198_V4.patch,
> HBASE-15198_V5.patch, HBASE-15198_V6.patch, HBASE-15198_V6.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142808#comment-15142808
]
Hudson commented on HBASE-15198:
SUCCESS: Integrated in HBase-1.3 #550 (See
[https://builds.apache.org/job/HBase-1.3/550/])
HBASE-15198 RPC client not using Codec and CellBlock for puts by (anoopsamjohn:
rev 19f8faeb8834db8b61f7c920ff2eef6734c1e593)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClient.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionAdapter.java
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142831#comment-15142831
]
Hadoop QA commented on HBASE-15198:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} HBASE-15198 does not apply to branch-1.0. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12787478/HBASE-15198_addendum-branch-1.0.patch
|
| JIRA Issue | HBASE-15198 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/518/console |
This message was automatically generated.
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch,
> HBASE-15198_addendum-branch-1.0.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142847#comment-15142847
]
Hudson commented on HBASE-15198:
SUCCESS: Integrated in HBase-1.3-IT #495 (See
[https://builds.apache.org/job/HBase-1.3-IT/495/])
HBASE-15198 RPC client not using Codec and CellBlock for puts by (anoopsamjohn:
rev 19f8faeb8834db8b61f7c920ff2eef6734c1e593)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionAdapter.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch,
> HBASE-15198_addendum-branch-1.0.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15252) Data loss when replaying wal if HDFS timeout
[
https://issues.apache.org/jira/browse/HBASE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142856#comment-15142856
]
Ted Yu edited comment on HBASE-15252 at 2/11/16 3:22 PM:
-
Edit:
The following test failure can be reproduced locally with patch:
{code}
Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 45.211 sec <<<
FAILURE! - in org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay
testDatalossWhenInputError(org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay)
Time elapsed: 0.352 sec <<< ERROR!
java.io.IOException: Got unknown writer class: SecureProtobufLogWriter
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:205)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:154)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:66)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:149)
at
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.testDatalossWhenInputError(TestWALReplay.java:978)
{code}
was (Author: yuzhih...@gmail.com):
Edit:
The following test failure can be reproduced locally with path:
{code}
Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 45.211 sec <<<
FAILURE! - in org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay
testDatalossWhenInputError(org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay)
Time elapsed: 0.352 sec <<< ERROR!
java.io.IOException: Got unknown writer class: SecureProtobufLogWriter
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:205)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:154)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:66)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:149)
at
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.testDatalossWhenInputError(TestWALReplay.java:978)
{code}
> Data loss when replaying wal if HDFS timeout
>
>
> Key: HBASE-15252
> URL: https://issues.apache.org/jira/browse/HBASE-15252
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15252-testcase.patch, HBASE-15252.patch
>
>
> This is a problem introduced by HBASE-13825 where we change the exception
> type in catch block in {{readNext}} method of {{ProtobufLogReader}}.
> {code:title=ProtobufLogReader.java}
> try {
> ..
> ProtobufUtil.mergeFrom(builder, new
> LimitInputStream(this.inputStream, size),
> (int)size);
> } catch (IOException ipbe) { // <-- used to be
> InvalidProtocolBufferException
> throw (EOFException) new EOFException("Invalid PB, EOF? Ignoring;
> originalPosition=" +
> originalPosition + ", currentPosition=" +
> this.inputStream.getPos() +
> ", messageSize=" + size + ", currentAvailable=" +
> available).initCause(ipbe);
> }
> {code}
> Here if the {{inputStream}} throws an {{IOException}} due to timeout or
> something, we just convert it to an {{EOFException}} and at the bottom of
> this method, we ignore {{EOFException}} and return false. This cause the
> upper layer think we reach the end of file. So when replaying we will treat
> the HDFS timeout error as a normal end of file and cause data loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15252) Data loss when replaying wal if HDFS timeout
[
https://issues.apache.org/jira/browse/HBASE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142856#comment-15142856
]
Ted Yu edited comment on HBASE-15252 at 2/11/16 3:22 PM:
-
Edit:
The following test failure can be reproduced locally with path:
{code}
Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 45.211 sec <<<
FAILURE! - in org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay
testDatalossWhenInputError(org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay)
Time elapsed: 0.352 sec <<< ERROR!
java.io.IOException: Got unknown writer class: SecureProtobufLogWriter
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:205)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:154)
at
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:66)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:149)
at
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.testDatalossWhenInputError(TestWALReplay.java:978)
{code}
was (Author: yuzhih...@gmail.com):
+1
> Data loss when replaying wal if HDFS timeout
>
>
> Key: HBASE-15252
> URL: https://issues.apache.org/jira/browse/HBASE-15252
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15252-testcase.patch, HBASE-15252.patch
>
>
> This is a problem introduced by HBASE-13825 where we change the exception
> type in catch block in {{readNext}} method of {{ProtobufLogReader}}.
> {code:title=ProtobufLogReader.java}
> try {
> ..
> ProtobufUtil.mergeFrom(builder, new
> LimitInputStream(this.inputStream, size),
> (int)size);
> } catch (IOException ipbe) { // <-- used to be
> InvalidProtocolBufferException
> throw (EOFException) new EOFException("Invalid PB, EOF? Ignoring;
> originalPosition=" +
> originalPosition + ", currentPosition=" +
> this.inputStream.getPos() +
> ", messageSize=" + size + ", currentAvailable=" +
> available).initCause(ipbe);
> }
> {code}
> Here if the {{inputStream}} throws an {{IOException}} due to timeout or
> something, we just convert it to an {{EOFException}} and at the bottom of
> this method, we ignore {{EOFException}} and return false. This cause the
> upper layer think we reach the end of file. So when replaying we will treat
> the HDFS timeout error as a normal end of file and cause data loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15249) Provide lower bound on number of regions in region normalizer for pre-split tables
[ https://issues.apache.org/jira/browse/HBASE-15249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142821#comment-15142821 ] Ted Yu commented on HBASE-15249: If some regions are empty, they wouldn't get write requests. But if regions receive even access, it is likely that regions would get write requests. How about giving regions whose write requests are in the bottom 10% (can make this configurable) ? > Provide lower bound on number of regions in region normalizer for pre-split > tables > -- > > Key: HBASE-15249 > URL: https://issues.apache.org/jira/browse/HBASE-15249 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: HBASE-15249.v1.txt, HBASE-15249.v2.txt > > > AMS (Ambari Metrics System) developer found the following scenario: > Metrics table was pre-split with many regions on large cluster (1600 nodes). > After some time, AMS stopped working because region normalizer merged the > regions into few big regions which were not able to serve high read / write > load. > This is a big problem since the write requests flood the regions faster than > the splits can happen resulting in poor performance. > We should consider setting reasonable lower bound on region count. > If the table is pre-split, we can use initial region count as the lower bound. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15252) Data loss when replaying wal if HDFS timeout
[
https://issues.apache.org/jira/browse/HBASE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142856#comment-15142856
]
Ted Yu commented on HBASE-15252:
+1
> Data loss when replaying wal if HDFS timeout
>
>
> Key: HBASE-15252
> URL: https://issues.apache.org/jira/browse/HBASE-15252
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15252-testcase.patch, HBASE-15252.patch
>
>
> This is a problem introduced by HBASE-13825 where we change the exception
> type in catch block in {{readNext}} method of {{ProtobufLogReader}}.
> {code:title=ProtobufLogReader.java}
> try {
> ..
> ProtobufUtil.mergeFrom(builder, new
> LimitInputStream(this.inputStream, size),
> (int)size);
> } catch (IOException ipbe) { // <-- used to be
> InvalidProtocolBufferException
> throw (EOFException) new EOFException("Invalid PB, EOF? Ignoring;
> originalPosition=" +
> originalPosition + ", currentPosition=" +
> this.inputStream.getPos() +
> ", messageSize=" + size + ", currentAvailable=" +
> available).initCause(ipbe);
> }
> {code}
> Here if the {{inputStream}} throws an {{IOException}} due to timeout or
> something, we just convert it to an {{EOFException}} and at the bottom of
> this method, we ignore {{EOFException}} and return false. This cause the
> upper layer think we reach the end of file. So when replaying we will treat
> the HDFS timeout error as a normal end of file and cause data loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10123) Change default ports; move them out of linux ephemeral port range
[ https://issues.apache.org/jira/browse/HBASE-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142862#comment-15142862 ] Jonathan Hsieh commented on HBASE-10123: Do you have a suggestion on what to do? The most obvious would be to have one of the systems change their default ports. I don't have evidence at the moment, but I'd be surprised if there are more Hadoop kms users than HBase users. Timeline wise, the changes were released at roughly same time -- This change was in a released by hbase in 30/oct/2014 [1] and in hadoop 18/nov/2014 [2]. [1] https://archive.apache.org/dist/hbase/hbase-0.99.0/ [2] http://hadoop.apache.org/releases.html > Change default ports; move them out of linux ephemeral port range > - > > Key: HBASE-10123 > URL: https://issues.apache.org/jira/browse/HBASE-10123 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.1.1 >Reporter: stack >Assignee: Jonathan Hsieh >Priority: Critical > Fix For: 0.99.0 > > Attachments: hbase-10123.patch, hbase-10123.v2.patch, > hbase-10123.v3.patch, hbase-10123.v4.patch > > > Our defaults clash w/ the range linux assigns itself for creating come-and-go > ephemeral ports; likely in our history we've clashed w/ a random, short-lived > process. While easy to change the defaults, we should just ship w/ defaults > that make sense. We could host ourselves up into the 7 or 8k range. > See http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-15198:
---
Attachment: HBASE-15198_addendum-branch-1.0.patch
TestAsyncProcess fix needed in branch-1.0 and branch-1.1 also.. Committed this
addendum to branches.
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch,
> HBASE-15198_addendum-branch-1.0.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15252) Data loss when replaying wal if HDFS timeout
[
https://issues.apache.org/jira/browse/HBASE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144062#comment-15144062
]
Hadoop QA commented on HBASE-15252:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 10s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 123m 35s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 105m 11s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 280m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_95 Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestBlockEvictionFromClient |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-12 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12787575/HBASE-15252-v1.patch |
| JIRA Issue | HBASE-15252 |
| Optional Tests | asflicense javac javadoc unit
[jira] [Commented] (HBASE-15252) Data loss when replaying wal if HDFS timeout
[
https://issues.apache.org/jira/browse/HBASE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144073#comment-15144073
]
Ted Yu commented on HBASE-15252:
Test failures don't seen to be related to patch.
+1
> Data loss when replaying wal if HDFS timeout
>
>
> Key: HBASE-15252
> URL: https://issues.apache.org/jira/browse/HBASE-15252
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15252-testcase.patch, HBASE-15252-v1.patch,
> HBASE-15252.patch
>
>
> This is a problem introduced by HBASE-13825 where we change the exception
> type in catch block in {{readNext}} method of {{ProtobufLogReader}}.
> {code:title=ProtobufLogReader.java}
> try {
> ..
> ProtobufUtil.mergeFrom(builder, new
> LimitInputStream(this.inputStream, size),
> (int)size);
> } catch (IOException ipbe) { // <-- used to be
> InvalidProtocolBufferException
> throw (EOFException) new EOFException("Invalid PB, EOF? Ignoring;
> originalPosition=" +
> originalPosition + ", currentPosition=" +
> this.inputStream.getPos() +
> ", messageSize=" + size + ", currentAvailable=" +
> available).initCause(ipbe);
> }
> {code}
> Here if the {{inputStream}} throws an {{IOException}} due to timeout or
> something, we just convert it to an {{EOFException}} and at the bottom of
> this method, we ignore {{EOFException}} and return false. This cause the
> upper layer think we reach the end of file. So when replaying we will treat
> the HDFS timeout error as a normal end of file and cause data loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14451) Move on to htrace-4.0.1 (from htrace-3.2.0) and tell a couple of good trace stories
[ https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143273#comment-15143273 ] Sean Busbey commented on HBASE-14451: - I think there's a decent argument to be made that HTrace 3.x stuff wasn't actually usable and the operational benefit to backporting is worth going earlier, but I think we'd need a [DISCUSS] on the matter. > Move on to htrace-4.0.1 (from htrace-3.2.0) and tell a couple of good trace > stories > --- > > Key: HBASE-14451 > URL: https://issues.apache.org/jira/browse/HBASE-14451 > Project: HBase > Issue Type: Task > Components: Operability, Performance >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: 14451.txt, 14451.v10.txt, 14451.v10.txt, 14451v11.patch, > 14451v13.txt, 14451v15.patch, 14451v2.txt, 14451v3.txt, 14451v4.txt, > 14451v5.txt, 14451v6.txt, 14451v7.txt, 14451v8.txt, 14451v9.txt, > 14551v12.patch, 14888v14.txt > > > htrace-4.0.0 was just release with a new API. Get up on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15151) turn off strict findbugs in precommit tests
[ https://issues.apache.org/jira/browse/HBASE-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143289#comment-15143289 ] Sean Busbey commented on HBASE-15151: - I didn't think all modules on all branches were at 0 yet. If that's true then maybe we can ignore this again. My understanding is that in non-strict mode we'll still get a result line that says there are extant findbugs problems, it'll just be with a vote of 0 and in blue (rather than -1 and red). I suspect this means the folks who would look out for existing problems will still notice it and everyone else will become blind to it. > turn off strict findbugs in precommit tests > --- > > Key: HBASE-15151 > URL: https://issues.apache.org/jira/browse/HBASE-15151 > Project: HBase > Issue Type: Task > Components: build, test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > > the "-1" for extant findbugs warnings has confused interpretation of our > precommit checks enough that we should switch to non-strict mode. > It will still record the number of findbugs warnings present before the > patch, but it'll vote "0" rather than calling attention to things via a -1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14451) Move on to htrace-4.0.1 (from htrace-3.2.0) and tell a couple of good trace stories
[
https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143299#comment-15143299
]
Hadoop QA commented on HBASE-14451:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 45s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 57s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 24m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 3m
11s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 8m 23s
{color} | {color:red} branch/. no findbugs output file
(./target/findbugsXml.xml) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 48s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 56s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 34s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 6s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 7s
{color} | {color:red} hbase-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 9s
{color} | {color:red} hbase-external-blockcache in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 7s
{color} | {color:red} hbase-it in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 9s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 7s
{color} | {color:red} hbase-shell in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 8s
{color} | {color:red} hbase-thrift in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 25s
{color} | {color:red} root in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 5s
{color} | {color:red} hbase-client in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 6s
{color} | {color:red} hbase-common in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 8s
{color} | {color:red} hbase-external-blockcache in the patch failed with JDK
v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 6s
{color} | {color:red} hbase-it in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 6s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 6s
{color} | {color:red} hbase-shell in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 6s
{color} | {color:red} hbase-thrift in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 25s {color} |
{color:red} root in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 5s {color} |
{color:red} hbase-client in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} |
[jira] [Commented] (HBASE-14451) Move on to htrace-4.0.1 (from htrace-3.2.0) and tell a couple of good trace stories
[ https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143353#comment-15143353 ] stack commented on HBASE-14451: --- [~busbey] Makes sense. Let me get something working first. Then will put up DISCUSS > Move on to htrace-4.0.1 (from htrace-3.2.0) and tell a couple of good trace > stories > --- > > Key: HBASE-14451 > URL: https://issues.apache.org/jira/browse/HBASE-14451 > Project: HBase > Issue Type: Task > Components: Operability, Performance >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: 14451.txt, 14451.v10.txt, 14451.v10.txt, 14451v11.patch, > 14451v13.txt, 14451v15.patch, 14451v2.txt, 14451v3.txt, 14451v4.txt, > 14451v5.txt, 14451v6.txt, 14451v7.txt, 14451v8.txt, 14451v9.txt, > 14551v12.patch, 14888v14.txt > > > htrace-4.0.0 was just release with a new API. Get up on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15129) Set default value for hbase.fs.tmp.dir rather than fully depend on hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143409#comment-15143409
]
Hudson commented on HBASE-15129:
SUCCESS: Integrated in HBase-1.2-IT #431 (See
[https://builds.apache.org/job/HBase-1.2-IT/431/])
HBASE-15129 Set default value for hbase.fs.tmp.dir rather than fully (ndimiduk:
rev 644326b65894d9eacde34434ea19063386c64618)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/security/SecureBulkLoadUtil.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
> Set default value for hbase.fs.tmp.dir rather than fully depend on
> hbase-default.xml
>
>
> Key: HBASE-15129
> URL: https://issues.apache.org/jira/browse/HBASE-15129
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15129.patch, HBASE-15129_v2.patch
>
>
> One of our users has observed below error when our cluster upgrades from
> 0.98.12 to 1.1.2:
> {noformat}
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122)
> at org.apache.hadoop.fs.Path.(Path.java:134)
> at org.apache.hadoop.fs.Path.(Path.java:88)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner(HFileOutputFormat2.java:639)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:489)
> {noformat}
> And checking the application code:
> {code}
> private Job createJob(Configuration jobConf) throws IOException {
> String tableName = jobConf.get(TABLE_NAME_KEY);
> Job job = new Job(jobConf);
> Configuration tableConf = HBaseConfiguration.create();
> HTable table = new HTable(tableConf, tableName);
> HFileOutputFormat2.configureIncrementalLoad(job, table);
> return job;
> }
> {code}
> We could see the user has his own hbase-independent job configuration, so our
> current code in HFileOutputFormat2:
> {code}
> Path partitionsPath = new Path(conf.get("hbase.fs.tmp.dir"), "partitions_" +
> UUID.randomUUID());
> {code}
> will return a null string and cause the above exception.
> We propose to add default value in the code rather than depend on
> hbase-default.xml in this JIRA.
> This is an improvement for HBASE-13625
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11927) Use Native Hadoop Library for HFile checksum (And flip default from CRC32 to CRC32C)
[
https://issues.apache.org/jira/browse/HBASE-11927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143293#comment-15143293
]
Hadoop QA commented on HBASE-11927:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} branch-1.1 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} branch-1.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 41s
{color} | {color:red} hbase-common in branch-1.1 has 9 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 46s
{color} | {color:red} hbase-server in branch-1.1 has 80 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 14s
{color} | {color:red} hbase-common in branch-1.1 failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 24s
{color} | {color:red} hbase-server in branch-1.1 failed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 4m
9s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s
{color} | {color:red} hbase-common in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 26s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 21s
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 46s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 29s
{color} | {color:green} hbase-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 86m 10s
{color} | {color:green} hbase-server in the patch
[jira] [Commented] (HBASE-15129) Set default value for hbase.fs.tmp.dir rather than fully depend on hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143451#comment-15143451
]
Hudson commented on HBASE-15129:
FAILURE: Integrated in HBase-1.2 #543 (See
[https://builds.apache.org/job/HBase-1.2/543/])
HBASE-15129 Set default value for hbase.fs.tmp.dir rather than fully (ndimiduk:
rev 644326b65894d9eacde34434ea19063386c64618)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/security/SecureBulkLoadUtil.java
> Set default value for hbase.fs.tmp.dir rather than fully depend on
> hbase-default.xml
>
>
> Key: HBASE-15129
> URL: https://issues.apache.org/jira/browse/HBASE-15129
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15129.patch, HBASE-15129_v2.patch
>
>
> One of our users has observed below error when our cluster upgrades from
> 0.98.12 to 1.1.2:
> {noformat}
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122)
> at org.apache.hadoop.fs.Path.(Path.java:134)
> at org.apache.hadoop.fs.Path.(Path.java:88)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner(HFileOutputFormat2.java:639)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:489)
> {noformat}
> And checking the application code:
> {code}
> private Job createJob(Configuration jobConf) throws IOException {
> String tableName = jobConf.get(TABLE_NAME_KEY);
> Job job = new Job(jobConf);
> Configuration tableConf = HBaseConfiguration.create();
> HTable table = new HTable(tableConf, tableName);
> HFileOutputFormat2.configureIncrementalLoad(job, table);
> return job;
> }
> {code}
> We could see the user has his own hbase-independent job configuration, so our
> current code in HFileOutputFormat2:
> {code}
> Path partitionsPath = new Path(conf.get("hbase.fs.tmp.dir"), "partitions_" +
> UUID.randomUUID());
> {code}
> will return a null string and cause the above exception.
> We propose to add default value in the code rather than depend on
> hbase-default.xml in this JIRA.
> This is an improvement for HBASE-13625
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15129) Set default value for hbase.fs.tmp.dir rather than fully depend on hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143285#comment-15143285
]
Andrew Purtell commented on HBASE-15129:
Definitely worthy of a release note.
> Set default value for hbase.fs.tmp.dir rather than fully depend on
> hbase-default.xml
>
>
> Key: HBASE-15129
> URL: https://issues.apache.org/jira/browse/HBASE-15129
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15129.patch, HBASE-15129_v2.patch
>
>
> One of our users has observed below error when our cluster upgrades from
> 0.98.12 to 1.1.2:
> {noformat}
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122)
> at org.apache.hadoop.fs.Path.(Path.java:134)
> at org.apache.hadoop.fs.Path.(Path.java:88)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner(HFileOutputFormat2.java:639)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:489)
> {noformat}
> And checking the application code:
> {code}
> private Job createJob(Configuration jobConf) throws IOException {
> String tableName = jobConf.get(TABLE_NAME_KEY);
> Job job = new Job(jobConf);
> Configuration tableConf = HBaseConfiguration.create();
> HTable table = new HTable(tableConf, tableName);
> HFileOutputFormat2.configureIncrementalLoad(job, table);
> return job;
> }
> {code}
> We could see the user has his own hbase-independent job configuration, so our
> current code in HFileOutputFormat2:
> {code}
> Path partitionsPath = new Path(conf.get("hbase.fs.tmp.dir"), "partitions_" +
> UUID.randomUUID());
> {code}
> will return a null string and cause the above exception.
> We propose to add default value in the code rather than depend on
> hbase-default.xml in this JIRA.
> This is an improvement for HBASE-13625
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15224) Undo "hbase.increment.fast.but.narrow.consistency" option; it is not necessary since HBASE-15213
[
https://issues.apache.org/jira/browse/HBASE-15224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143367#comment-15143367
]
Hadoop QA commented on HBASE-15224:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 4m
9s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 23s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 16s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 181m 1s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.hbase.mapreduce.TestImportExport |
| JDK v1.7.0_95 Failed junit tests | hadoop.hbase.mapreduce.TestImportExport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-11 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12787343/15224.branch-1.2.addendum.patch
|
| JIRA Issue | HBASE-15224 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux b3700cc81157
[jira] [Commented] (HBASE-14451) Move on to htrace-4.0.1 (from htrace-3.2.0) and tell a couple of good trace stories
[ https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143269#comment-15143269 ] stack commented on HBASE-14451: --- [~busbey] I was going to hack on a 1.2. I suppose 1.3? > Move on to htrace-4.0.1 (from htrace-3.2.0) and tell a couple of good trace > stories > --- > > Key: HBASE-14451 > URL: https://issues.apache.org/jira/browse/HBASE-14451 > Project: HBase > Issue Type: Task > Components: Operability, Performance >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: 14451.txt, 14451.v10.txt, 14451.v10.txt, 14451v11.patch, > 14451v13.txt, 14451v15.patch, 14451v2.txt, 14451v3.txt, 14451v4.txt, > 14451v5.txt, 14451v6.txt, 14451v7.txt, 14451v8.txt, 14451v9.txt, > 14551v12.patch, 14888v14.txt > > > htrace-4.0.0 was just release with a new API. Get up on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11927) Use Native Hadoop Library for HFile checksum (And flip default from CRC32 to CRC32C)
[ https://issues.apache.org/jira/browse/HBASE-11927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143291#comment-15143291 ] Andrew Purtell commented on HBASE-11927: +1 I made a similar assessment about 0.98, hence HBASE-14738, but have held off commit because of the rather extensive additional changes needed for 0.98-specific complications of dealing with multiple versions of Hadoop. > Use Native Hadoop Library for HFile checksum (And flip default from CRC32 to > CRC32C) > > > Key: HBASE-11927 > URL: https://issues.apache.org/jira/browse/HBASE-11927 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: Appy > Fix For: 2.0.0, 1.2.0, 1.1.4 > > Attachments: HBASE-11927-branch-1.1.patch, HBASE-11927-v1.patch, > HBASE-11927-v2.patch, HBASE-11927-v4.patch, HBASE-11927-v5.patch, > HBASE-11927-v6.patch, HBASE-11927-v7.patch, HBASE-11927-v8.patch, > HBASE-11927-v8.patch, HBASE-11927.patch, after-compact-2%.svg, > after-randomWrite1M-0.5%.svg, before-compact-22%.svg, > before-randomWrite1M-5%.svg, c2021.crc2.svg, c2021.write.2.svg, > c2021.zip.svg, crc32ct.svg > > > Up in hadoop they have this change. Let me publish some graphs to show that > it makes a difference (CRC is a massive amount of our CPU usage in my > profiling of an upload because of compacting, flushing, etc.). We should > also make use of native CRCings -- especially the 2.6 HDFS-6865 and ilk -- in > hbase but that is another issue for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11927) Use Native Hadoop Library for HFile checksum (And flip default from CRC32 to CRC32C)
[ https://issues.apache.org/jira/browse/HBASE-11927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143358#comment-15143358 ] stack commented on HBASE-11927: --- Backport looks good to me. You'll need to call out in any release the flip to CRC32C (it won't be a problem but a change). Its a nice boost so worth the backport. > Use Native Hadoop Library for HFile checksum (And flip default from CRC32 to > CRC32C) > > > Key: HBASE-11927 > URL: https://issues.apache.org/jira/browse/HBASE-11927 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Assignee: Appy > Fix For: 2.0.0, 1.2.0, 1.1.4 > > Attachments: HBASE-11927-branch-1.1.patch, HBASE-11927-v1.patch, > HBASE-11927-v2.patch, HBASE-11927-v4.patch, HBASE-11927-v5.patch, > HBASE-11927-v6.patch, HBASE-11927-v7.patch, HBASE-11927-v8.patch, > HBASE-11927-v8.patch, HBASE-11927.patch, after-compact-2%.svg, > after-randomWrite1M-0.5%.svg, before-compact-22%.svg, > before-randomWrite1M-5%.svg, c2021.crc2.svg, c2021.write.2.svg, > c2021.zip.svg, crc32ct.svg > > > Up in hadoop they have this change. Let me publish some graphs to show that > it makes a difference (CRC is a massive amount of our CPU usage in my > profiling of an upload because of compacting, flushing, etc.). We should > also make use of native CRCings -- especially the 2.6 HDFS-6865 and ilk -- in > hbase but that is another issue for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15224) Undo "hbase.increment.fast.but.narrow.consistency" option; it is not necessary since HBASE-15213
[ https://issues.apache.org/jira/browse/HBASE-15224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143387#comment-15143387 ] Elliott Clark commented on HBASE-15224: --- +1 on the addendum > Undo "hbase.increment.fast.but.narrow.consistency" option; it is not > necessary since HBASE-15213 > -- > > Key: HBASE-15224 > URL: https://issues.apache.org/jira/browse/HBASE-15224 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack > Fix For: 1.2.1, 1.1.4, 1.0.4 > > Attachments: > 0001-HBASE-15224-Undo-hbase.increment.fast.branch-1.2.patch, > 0001-HBASE-15224-branch-1.2.patch, 15224.branch-1.2.addendum.patch, > 15224.branch-1.2.patch, HBASE-15224.branch-1.2.patch, > HBASE-15224v2-branch-1.2.patch > > > Remove an escape valve no longer needed after HBASE-15213. Remove so folks > don't ever have to worry their pretty-little heads about it. > Let the bulk of HBASE-15031 remain, the issue that added > "hbase.increment.fast.but.narrow.consistency" because it mostly cleanup we > want to keep. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15252) Data loss when replaying wal if HDFS timeout
[
https://issues.apache.org/jira/browse/HBASE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-15252:
--
Attachment: HBASE-15252.patch
So let's fix this issue first.
The readNext thing could be talked at another issue.
> Data loss when replaying wal if HDFS timeout
>
>
> Key: HBASE-15252
> URL: https://issues.apache.org/jira/browse/HBASE-15252
> Project: HBase
> Issue Type: Bug
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-15252-testcase.patch, HBASE-15252.patch
>
>
> This is a problem introduced by HBASE-13825 where we change the exception
> type in catch block in {{readNext}} method of {{ProtobufLogReader}}.
> {code:title=ProtobufLogReader.java}
> try {
> ..
> ProtobufUtil.mergeFrom(builder, new
> LimitInputStream(this.inputStream, size),
> (int)size);
> } catch (IOException ipbe) { // <-- used to be
> InvalidProtocolBufferException
> throw (EOFException) new EOFException("Invalid PB, EOF? Ignoring;
> originalPosition=" +
> originalPosition + ", currentPosition=" +
> this.inputStream.getPos() +
> ", messageSize=" + size + ", currentAvailable=" +
> available).initCause(ipbe);
> }
> {code}
> Here if the {{inputStream}} throws an {{IOException}} due to timeout or
> something, we just convert it to an {{EOFException}} and at the bottom of
> this method, we ignore {{EOFException}} and return false. This cause the
> upper layer think we reach the end of file. So when replaying we will treat
> the HDFS timeout error as a normal end of file and cause data loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142504#comment-15142504
]
Hadoop QA commented on HBASE-15198:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
20s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} branch-1 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} branch-1 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
15s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} branch-1 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} branch-1 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 4m
32s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 37s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 88m 38s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 41s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 46s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 210m 17s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_95 Failed junit tests | hadoop.hbase.mapreduce.TestImportExport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-11 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142619#comment-15142619
]
Anoop Sam John commented on HBASE-15198:
Committed to branch-1.0+ branches.
Committed to 0.98 also but got reproducible failures and so reverted. Checking
it.
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-branch-1.0.patch,
> HBASE-15198-branch-1.1.patch, HBASE-15198-branch-1.patch, HBASE-15198.patch,
> HBASE-15198_V2.patch, HBASE-15198_V3.patch, HBASE-15198_V4.patch,
> HBASE-15198_V5.patch, HBASE-15198_V6.patch, HBASE-15198_V6.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142640#comment-15142640
]
Hudson commented on HBASE-15198:
FAILURE: Integrated in HBase-1.2 #542 (See
[https://builds.apache.org/job/HBase-1.2/542/])
HBASE-15198 RPC client not using Codec and CellBlock for puts by (anoopsamjohn:
rev 7d5a158c0bf5cf497917182e72195dba6be11afc)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClient.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionAdapter.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-branch-1.0.patch,
> HBASE-15198-branch-1.1.patch, HBASE-15198-branch-1.patch, HBASE-15198.patch,
> HBASE-15198_V2.patch, HBASE-15198_V3.patch, HBASE-15198_V4.patch,
> HBASE-15198_V5.patch, HBASE-15198_V6.patch, HBASE-15198_V6.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15252) Data loss when replaying wal if HDFS timeout
[
https://issues.apache.org/jira/browse/HBASE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-15252:
--
Fix Version/s: 0.98.18
1.0.4
1.1.4
1.2.1
1.3.0
2.0.0
Affects Version/s: 1.3.0
1.2.0
2.0.0
1.0.3
1.1.3
0.98.17
Status: Patch Available (was: Open)
> Data loss when replaying wal if HDFS timeout
>
>
> Key: HBASE-15252
> URL: https://issues.apache.org/jira/browse/HBASE-15252
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 0.98.17, 1.1.3, 1.0.3, 2.0.0, 1.2.0, 1.3.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15252-testcase.patch, HBASE-15252.patch
>
>
> This is a problem introduced by HBASE-13825 where we change the exception
> type in catch block in {{readNext}} method of {{ProtobufLogReader}}.
> {code:title=ProtobufLogReader.java}
> try {
> ..
> ProtobufUtil.mergeFrom(builder, new
> LimitInputStream(this.inputStream, size),
> (int)size);
> } catch (IOException ipbe) { // <-- used to be
> InvalidProtocolBufferException
> throw (EOFException) new EOFException("Invalid PB, EOF? Ignoring;
> originalPosition=" +
> originalPosition + ", currentPosition=" +
> this.inputStream.getPos() +
> ", messageSize=" + size + ", currentAvailable=" +
> available).initCause(ipbe);
> }
> {code}
> Here if the {{inputStream}} throws an {{IOException}} due to timeout or
> something, we just convert it to an {{EOFException}} and at the bottom of
> this method, we ignore {{EOFException}} and return false. This cause the
> upper layer think we reach the end of file. So when replaying we will treat
> the HDFS timeout error as a normal end of file and cause data loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142590#comment-15142590
]
Hadoop QA commented on HBASE-15198:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} 0.98 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} 0.98 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} 0.98 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 53s
{color} | {color:red} hbase-client in 0.98 has 19 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 39s
{color} | {color:red} hbase-server in 0.98 has 84 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 18s
{color} | {color:red} hbase-client in 0.98 failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s
{color} | {color:red} hbase-server in 0.98 failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} 0.98 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 3m
26s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 19s
{color} | {color:red} hbase-client in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 27s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 8s {color} |
{color:red} hbase-client in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 56s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 17s {color}
| {color:red} hbase-client in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 6s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142630#comment-15142630
]
Hudson commented on HBASE-15198:
SUCCESS: Integrated in HBase-1.2-IT #430 (See
[https://builds.apache.org/job/HBase-1.2-IT/430/])
HBASE-15198 RPC client not using Codec and CellBlock for puts by (anoopsamjohn:
rev 7d5a158c0bf5cf497917182e72195dba6be11afc)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionAdapter.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-branch-1.0.patch,
> HBASE-15198-branch-1.1.patch, HBASE-15198-branch-1.patch, HBASE-15198.patch,
> HBASE-15198_V2.patch, HBASE-15198_V3.patch, HBASE-15198_V4.patch,
> HBASE-15198_V5.patch, HBASE-15198_V6.patch, HBASE-15198_V6.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15253) Small bug in CellUtil.matchingRow(Cell, byte[])
[
https://issues.apache.org/jira/browse/HBASE-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142720#comment-15142720
]
Anoop Sam John commented on HBASE-15253:
+1
> Small bug in CellUtil.matchingRow(Cell, byte[])
> ---
>
> Key: HBASE-15253
> URL: https://issues.apache.org/jira/browse/HBASE-15253
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15253.patch
>
>
> {code}
> public static boolean matchingRow(final Cell left, final byte[] buf) {
> if (buf == null) {
> return left.getQualifierLength() == 0;
> }
> return matchingRow(left, buf, 0, buf.length);
> }
> {code}
> The 'if' condition should check for left.getRowLength() == 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142748#comment-15142748
]
Hudson commented on HBASE-15198:
FAILURE: Integrated in HBase-1.1-JDK8 #1746 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1746/])
HBASE-15198 RPC client not using Codec and CellBlock for puts by (anoopsamjohn:
rev bd11ae58bf05996952d960fbe3cbf743f1d781e5)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionAdapter.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClient.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-branch-1.0.patch,
> HBASE-15198-branch-1.1.patch, HBASE-15198-branch-1.patch, HBASE-15198.patch,
> HBASE-15198_V2.patch, HBASE-15198_V3.patch, HBASE-15198_V4.patch,
> HBASE-15198_V5.patch, HBASE-15198_V6.patch, HBASE-15198_V6.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15252) Data loss when replaying wal if HDFS timeout
[
https://issues.apache.org/jira/browse/HBASE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142771#comment-15142771
]
Hadoop QA commented on HBASE-15252:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 5s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 6s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 31s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 251m 34s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests |
hadoop.hbase.mob.compactions.TestMobCompactor |
| | hadoop.hbase.regionserver.wal.TestSecureWALReplay |
| JDK v1.8.0_72 Timed out junit tests |
org.apache.hadoop.hbase.TestClusterBootOrder |
| | org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
| JDK v1.7.0_95 Failed junit tests |
hadoop.hbase.regionserver.wal.TestSecureWALReplay |
| JDK v1.7.0_95 Timed out junit tests |
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
\\
\\
|| Subsystem || Report/Notes ||
|
[jira] [Commented] (HBASE-15252) Data loss when replaying wal if HDFS timeout
[
https://issues.apache.org/jira/browse/HBASE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142772#comment-15142772
]
stack commented on HBASE-15252:
---
+1 Nice test.
> Data loss when replaying wal if HDFS timeout
>
>
> Key: HBASE-15252
> URL: https://issues.apache.org/jira/browse/HBASE-15252
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15252-testcase.patch, HBASE-15252.patch
>
>
> This is a problem introduced by HBASE-13825 where we change the exception
> type in catch block in {{readNext}} method of {{ProtobufLogReader}}.
> {code:title=ProtobufLogReader.java}
> try {
> ..
> ProtobufUtil.mergeFrom(builder, new
> LimitInputStream(this.inputStream, size),
> (int)size);
> } catch (IOException ipbe) { // <-- used to be
> InvalidProtocolBufferException
> throw (EOFException) new EOFException("Invalid PB, EOF? Ignoring;
> originalPosition=" +
> originalPosition + ", currentPosition=" +
> this.inputStream.getPos() +
> ", messageSize=" + size + ", currentAvailable=" +
> available).initCause(ipbe);
> }
> {code}
> Here if the {{inputStream}} throws an {{IOException}} due to timeout or
> something, we just convert it to an {{EOFException}} and at the bottom of
> this method, we ignore {{EOFException}} and return false. This cause the
> upper layer think we reach the end of file. So when replaying we will treat
> the HDFS timeout error as a normal end of file and cause data loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15253) Small bug in CellUtil.matchingRow(Cell, byte[])
[
https://issues.apache.org/jira/browse/HBASE-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-15253:
---
Attachment: HBASE-15253.patch
Straight forward patch.
> Small bug in CellUtil.matchingRow(Cell, byte[])
> ---
>
> Key: HBASE-15253
> URL: https://issues.apache.org/jira/browse/HBASE-15253
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15253.patch
>
>
> {code}
> public static boolean matchingRow(final Cell left, final byte[] buf) {
> if (buf == null) {
> return left.getQualifierLength() == 0;
> }
> return matchingRow(left, buf, 0, buf.length);
> }
> {code}
> The 'if' condition should check for left.getRowLength() == 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15253) Small bug in CellUtil.matchingRow(Cell, byte[])
[
https://issues.apache.org/jira/browse/HBASE-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-15253:
---
Status: Patch Available (was: Open)
> Small bug in CellUtil.matchingRow(Cell, byte[])
> ---
>
> Key: HBASE-15253
> URL: https://issues.apache.org/jira/browse/HBASE-15253
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15253.patch
>
>
> {code}
> public static boolean matchingRow(final Cell left, final byte[] buf) {
> if (buf == null) {
> return left.getQualifierLength() == 0;
> }
> return matchingRow(left, buf, 0, buf.length);
> }
> {code}
> The 'if' condition should check for left.getRowLength() == 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15253) Small bug in CellUtil.matchingRow(Cell, byte[])
[
https://issues.apache.org/jira/browse/HBASE-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142775#comment-15142775
]
Hadoop QA commented on HBASE-15253:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 57s
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 42s
{color} | {color:green} hbase-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 43s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-11 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12787463/HBASE-15253.patch |
| JIRA Issue | HBASE-15253 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d5af68a687b1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142774#comment-15142774
]
Hudson commented on HBASE-15198:
FAILURE: Integrated in HBase-1.1-JDK7 #1659 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1659/])
HBASE-15198 RPC client not using Codec and CellBlock for puts by (anoopsamjohn:
rev bd11ae58bf05996952d960fbe3cbf743f1d781e5)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClient.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionAdapter.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-15198:
---
Attachment: HBASE-15198-0.98.patch
Failure in TestAsyncProcess because of not making required change in the test
class. It uses custom connection impl.
Failure in TestOpenTableInCoprocessor is because of an existing bug in the code
I can say. When some one added the codec explicitly in xml file,
MultiServerCallable will use cell blocks.
In CoprocessorHConnection doing callBlockingMethod, we pass null as the
CellScanner which will fail the op. When the PayloadCarryingRpcController
having some CellScanner with it, we have to pass that. Will fix in this patch
Failure in TestMultiParallel I can reproduce with out this patch. When I
committed HBASE-15214 ran that and was passing. Now also going to that commit
the test passing. The fail is because of some commit btw
38ce707b2706c1c48b4f2386cc679f14081fc523 and
b3fe2556641ad5b36caded1a995f09443d88ab5d. FYI [~apurtell] Let me check which
commit broke this
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15249) Provide lower bound on number of regions in region normalizer for pre-split tables
[ https://issues.apache.org/jira/browse/HBASE-15249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142935#comment-15142935 ] stack commented on HBASE-15249: --- bq. How about giving regions whose write requests are in the bottom 10% (can make this configurable) ? Above is totally arbitrary. If a region is getting 100 hits a second, thats 10% of 1k -- you'd merge it because it is getting too little load? bq. After some time, AMS stopped working because region normalizer merged the regions into few big regions which were not able to serve high read / write load. This is a big problem since the write requests flood the regions faster than the splits can happen resulting in poor performance. Numbers? How many regions was too little? What was the hit rate that overwhelmed? What did the normalizer do? It merged up how many regions in how much time? The problem here is that the time between presplit and the load coming on was too long? The normalizer merged up the regions before the load came on? > Provide lower bound on number of regions in region normalizer for pre-split > tables > -- > > Key: HBASE-15249 > URL: https://issues.apache.org/jira/browse/HBASE-15249 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: HBASE-15249.v1.txt, HBASE-15249.v2.txt > > > AMS (Ambari Metrics System) developer found the following scenario: > Metrics table was pre-split with many regions on large cluster (1600 nodes). > After some time, AMS stopped working because region normalizer merged the > regions into few big regions which were not able to serve high read / write > load. > This is a big problem since the write requests flood the regions faster than > the splits can happen resulting in poor performance. > We should consider setting reasonable lower bound on region count. > If the table is pre-split, we can use initial region count as the lower bound. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15252) Data loss when replaying wal if HDFS timeout
[
https://issues.apache.org/jira/browse/HBASE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142912#comment-15142912
]
Ted Yu commented on HBASE-15252:
TestWALReplay does this:
{code}
conf.setClass("hbase.regionserver.hlog.reader.impl",
SecureProtobufLogReader.class,
Reader.class);
{code}
In the new test:
{code}
ProtobufLogReader reader = new ProtobufLogReader();
{code}
Please retrieve value for "hbase.regionserver.hlog.reader.impl" and instantiate
the class instead of hardcode Reader class.
> Data loss when replaying wal if HDFS timeout
>
>
> Key: HBASE-15252
> URL: https://issues.apache.org/jira/browse/HBASE-15252
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15252-testcase.patch, HBASE-15252.patch
>
>
> This is a problem introduced by HBASE-13825 where we change the exception
> type in catch block in {{readNext}} method of {{ProtobufLogReader}}.
> {code:title=ProtobufLogReader.java}
> try {
> ..
> ProtobufUtil.mergeFrom(builder, new
> LimitInputStream(this.inputStream, size),
> (int)size);
> } catch (IOException ipbe) { // <-- used to be
> InvalidProtocolBufferException
> throw (EOFException) new EOFException("Invalid PB, EOF? Ignoring;
> originalPosition=" +
> originalPosition + ", currentPosition=" +
> this.inputStream.getPos() +
> ", messageSize=" + size + ", currentAvailable=" +
> available).initCause(ipbe);
> }
> {code}
> Here if the {{inputStream}} throws an {{IOException}} due to timeout or
> something, we just convert it to an {{EOFException}} and at the bottom of
> this method, we ignore {{EOFException}} and return false. This cause the
> upper layer think we reach the end of file. So when replaying we will treat
> the HDFS timeout error as a normal end of file and cause data loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15253) Small bug in CellUtil.matchingRow(Cell, byte[])
[
https://issues.apache.org/jira/browse/HBASE-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-15253:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to trunk. Thanks for the reviews.
> Small bug in CellUtil.matchingRow(Cell, byte[])
> ---
>
> Key: HBASE-15253
> URL: https://issues.apache.org/jira/browse/HBASE-15253
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15253.patch
>
>
> {code}
> public static boolean matchingRow(final Cell left, final byte[] buf) {
> if (buf == null) {
> return left.getQualifierLength() == 0;
> }
> return matchingRow(left, buf, 0, buf.length);
> }
> {code}
> The 'if' condition should check for left.getRowLength() == 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15251) During a cluster restart, Hmaster thinks it is a failover by mistake
[ https://issues.apache.org/jira/browse/HBASE-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Clara Xiong updated HBASE-15251: Attachment: HBASE-15251-master.patch > During a cluster restart, Hmaster thinks it is a failover by mistake > > > Key: HBASE-15251 > URL: https://issues.apache.org/jira/browse/HBASE-15251 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.0.0, 0.98.15 >Reporter: Clara Xiong >Assignee: Clara Xiong > Attachments: HBASE-15251-master.patch > > > We often need to do cluster restart as part of release for a cluster of > > 1000 nodes. We have tried our best to get clean shutdown but 50% of the time, > hmaster still thinks it is a failover. This increases the restart time from 5 > min to 30 min and decreases locality from 99% to 5% since we didn't use a > balancer. We had a bug HBASE-14129 but the fix didn't work. > After adding more logging and inspecting the logs, we identified two things > that trigger the failover handling: > 1. When Hmaster.AssignmentManager detects any dead servers on service > manager during joinCluster(), it determines this is a failover without > further check. I added a check whether there is even any region assigned to > these servers. During a clean restart, the regions are not even assigned. > 2. When there are some leftover empty folders for log and split directories > or empty wal files, it is also treated as a failover. I added a check for > that. Although this can be resolved by manual cleanup, it is still too > tedious for restarting a large cluster. > Patch will follow shortly. The fix is tested and used in production now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15245) Port HBASE-15177 (Reduce garbage created under high load) to 0.98
[
https://issues.apache.org/jira/browse/HBASE-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143640#comment-15143640
]
Hadoop QA commented on HBASE-15245:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
18s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} 0.98 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} 0.98 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s {color} | {color:green} 0.98 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 58s
{color} | {color:red} hbase-client in 0.98 has 19 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 41s
{color} | {color:red} hbase-common in 0.98 has 7 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 46s
{color} | {color:red} hbase-server in 0.98 has 84 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 19s
{color} | {color:red} hbase-client in 0.98 failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 15s
{color} | {color:red} hbase-common in 0.98 failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 27s
{color} | {color:red} hbase-server in 0.98 failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} 0.98 passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 15s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 25s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 21s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 45s {color}
| {color:red} hbase-common-jdk1.8.0_72 with JDK v1.8.0_72 generated 9 new
issues (was 6, now 11). {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 24s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 42s {color}
| {color:red} hbase-common-jdk1.7.0_95 with JDK v1.7.0_95 generated 9 new
issues (was 6, now 11). {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 24s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-common
(total was 11, now 12). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 3m
57s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 17s
{color} | {color:red} hbase-client in the patch failed with JDK v1.8.0_72.
{color} |
|
[jira] [Commented] (HBASE-15224) Undo "hbase.increment.fast.but.narrow.consistency" option; it is not necessary since HBASE-15213
[ https://issues.apache.org/jira/browse/HBASE-15224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143751#comment-15143751 ] Hudson commented on HBASE-15224: FAILURE: Integrated in HBase-1.2 #544 (See [https://builds.apache.org/job/HBase-1.2/544/]) HBASE-15224 Undo "hbase.increment.fast.but.narrow.consistency" option; (stack: rev cc863f37d4b8dd4f6c7a736abe8f2160c9b12ea2) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestIncrementFromClientSideWithCoprocessor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestIncrementsFromClientSide.java > Undo "hbase.increment.fast.but.narrow.consistency" option; it is not > necessary since HBASE-15213 > -- > > Key: HBASE-15224 > URL: https://issues.apache.org/jira/browse/HBASE-15224 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack > Fix For: 1.2.1, 1.1.4, 1.0.4 > > Attachments: > 0001-HBASE-15224-Undo-hbase.increment.fast.branch-1.2.patch, > 0001-HBASE-15224-branch-1.2.patch, 15224.branch-1.2.addendum.patch, > 15224.branch-1.2.patch, HBASE-15224.branch-1.2.patch, > HBASE-15224v2-branch-1.2.patch > > > Remove an escape valve no longer needed after HBASE-15213. Remove so folks > don't ever have to worry their pretty-little heads about it. > Let the bulk of HBASE-15031 remain, the issue that added > "hbase.increment.fast.but.narrow.consistency" because it mostly cleanup we > want to keep. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15129) Set default value for hbase.fs.tmp.dir rather than fully depend on hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143757#comment-15143757
]
Hudson commented on HBASE-15129:
FAILURE: Integrated in HBase-1.1-JDK8 #1747 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1747/])
HBASE-15129 Set default value for hbase.fs.tmp.dir rather than fully (ndimiduk:
rev 93c0764c5cff9928ef8703c58e1bc6e4dad5b71a)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/security/SecureBulkLoadUtil.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
> Set default value for hbase.fs.tmp.dir rather than fully depend on
> hbase-default.xml
>
>
> Key: HBASE-15129
> URL: https://issues.apache.org/jira/browse/HBASE-15129
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15129.patch, HBASE-15129_v2.patch
>
>
> One of our users has observed below error when our cluster upgrades from
> 0.98.12 to 1.1.2:
> {noformat}
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122)
> at org.apache.hadoop.fs.Path.(Path.java:134)
> at org.apache.hadoop.fs.Path.(Path.java:88)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner(HFileOutputFormat2.java:639)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:489)
> {noformat}
> And checking the application code:
> {code}
> private Job createJob(Configuration jobConf) throws IOException {
> String tableName = jobConf.get(TABLE_NAME_KEY);
> Job job = new Job(jobConf);
> Configuration tableConf = HBaseConfiguration.create();
> HTable table = new HTable(tableConf, tableName);
> HFileOutputFormat2.configureIncrementalLoad(job, table);
> return job;
> }
> {code}
> We could see the user has his own hbase-independent job configuration, so our
> current code in HFileOutputFormat2:
> {code}
> Path partitionsPath = new Path(conf.get("hbase.fs.tmp.dir"), "partitions_" +
> UUID.randomUUID());
> {code}
> will return a null string and cause the above exception.
> We propose to add default value in the code rather than depend on
> hbase-default.xml in this JIRA.
> This is an improvement for HBASE-13625
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15224) Undo "hbase.increment.fast.but.narrow.consistency" option; it is not necessary since HBASE-15213
[ https://issues.apache.org/jira/browse/HBASE-15224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143727#comment-15143727 ] Hudson commented on HBASE-15224: SUCCESS: Integrated in HBase-1.2-IT #432 (See [https://builds.apache.org/job/HBase-1.2-IT/432/]) HBASE-15224 Undo "hbase.increment.fast.but.narrow.consistency" option; (stack: rev cc863f37d4b8dd4f6c7a736abe8f2160c9b12ea2) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestIncrementFromClientSideWithCoprocessor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestIncrementsFromClientSide.java > Undo "hbase.increment.fast.but.narrow.consistency" option; it is not > necessary since HBASE-15213 > -- > > Key: HBASE-15224 > URL: https://issues.apache.org/jira/browse/HBASE-15224 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack > Fix For: 1.2.1, 1.1.4, 1.0.4 > > Attachments: > 0001-HBASE-15224-Undo-hbase.increment.fast.branch-1.2.patch, > 0001-HBASE-15224-branch-1.2.patch, 15224.branch-1.2.addendum.patch, > 15224.branch-1.2.patch, HBASE-15224.branch-1.2.patch, > HBASE-15224v2-branch-1.2.patch > > > Remove an escape valve no longer needed after HBASE-15213. Remove so folks > don't ever have to worry their pretty-little heads about it. > Let the bulk of HBASE-15031 remain, the issue that added > "hbase.increment.fast.but.narrow.consistency" because it mostly cleanup we > want to keep. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143755#comment-15143755
]
Hudson commented on HBASE-15198:
FAILURE: Integrated in HBase-1.1-JDK8 #1747 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1747/])
HBASE-15198 RPC client not using Codec and CellBlock for puts by (anoopsamjohn:
rev 29b6b2ccca67e6c183e5226ec0109c58840675df)
*
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch,
> HBASE-15198_addendum-branch-1.0.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15245) Port HBASE-15177 (Reduce garbage created under high load) to 0.98
[
https://issues.apache.org/jira/browse/HBASE-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143769#comment-15143769
]
Hadoop QA commented on HBASE-15245:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
55s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} 0.98 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} 0.98 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} 0.98 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 52s
{color} | {color:red} hbase-client in 0.98 has 19 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 40s
{color} | {color:red} hbase-common in 0.98 has 7 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 40s
{color} | {color:red} hbase-server in 0.98 has 84 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 21s
{color} | {color:red} hbase-client in 0.98 failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 17s
{color} | {color:red} hbase-common in 0.98 failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s
{color} | {color:red} hbase-server in 0.98 failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} 0.98 passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 13s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 22s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 20s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 32s {color}
| {color:red} hbase-common-jdk1.8.0_72 with JDK v1.8.0_72 generated 8 new
issues (was 6, now 11). {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 20s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 21s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 23s {color}
| {color:red} hbase-common-jdk1.7.0_95 with JDK v1.7.0_95 generated 8 new
issues (was 6, now 11). {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 3m
29s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 15s
{color} | {color:red} hbase-client in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
[jira] [Updated] (HBASE-15252) Data loss when replaying wal if HDFS timeout
[
https://issues.apache.org/jira/browse/HBASE-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-15252:
--
Attachment: HBASE-15252-v1.patch
New patch addressing the secure log reader issue.
> Data loss when replaying wal if HDFS timeout
>
>
> Key: HBASE-15252
> URL: https://issues.apache.org/jira/browse/HBASE-15252
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15252-testcase.patch, HBASE-15252-v1.patch,
> HBASE-15252.patch
>
>
> This is a problem introduced by HBASE-13825 where we change the exception
> type in catch block in {{readNext}} method of {{ProtobufLogReader}}.
> {code:title=ProtobufLogReader.java}
> try {
> ..
> ProtobufUtil.mergeFrom(builder, new
> LimitInputStream(this.inputStream, size),
> (int)size);
> } catch (IOException ipbe) { // <-- used to be
> InvalidProtocolBufferException
> throw (EOFException) new EOFException("Invalid PB, EOF? Ignoring;
> originalPosition=" +
> originalPosition + ", currentPosition=" +
> this.inputStream.getPos() +
> ", messageSize=" + size + ", currentAvailable=" +
> available).initCause(ipbe);
> }
> {code}
> Here if the {{inputStream}} throws an {{IOException}} due to timeout or
> something, we just convert it to an {{EOFException}} and at the bottom of
> this method, we ignore {{EOFException}} and return false. This cause the
> upper layer think we reach the end of file. So when replaying we will treat
> the HDFS timeout error as a normal end of file and cause data loss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143791#comment-15143791
]
stack commented on HBASE-15198:
---
[~anoop.hbase] This causes NPE in TestAsyncProcess on 1.2:
https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/542/
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch,
> HBASE-15198_addendum-branch-1.0.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143812#comment-15143812
]
Hudson commented on HBASE-15198:
SUCCESS: Integrated in HBase-1.1-JDK7 #1660 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1660/])
HBASE-15198 RPC client not using Codec and CellBlock for puts by (anoopsamjohn:
rev 29b6b2ccca67e6c183e5226ec0109c58840675df)
*
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198-0.98.patch, HBASE-15198-0.98.patch,
> HBASE-15198-branch-1.0.patch, HBASE-15198-branch-1.1.patch,
> HBASE-15198-branch-1.patch, HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch,
> HBASE-15198_V6.patch, HBASE-15198_V6.patch,
> HBASE-15198_addendum-branch-1.0.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15129) Set default value for hbase.fs.tmp.dir rather than fully depend on hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143813#comment-15143813
]
Hudson commented on HBASE-15129:
SUCCESS: Integrated in HBase-1.1-JDK7 #1660 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1660/])
HBASE-15129 Set default value for hbase.fs.tmp.dir rather than fully (ndimiduk:
rev 93c0764c5cff9928ef8703c58e1bc6e4dad5b71a)
*
hbase-client/src/main/java/org/apache/hadoop/hbase/security/SecureBulkLoadUtil.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
> Set default value for hbase.fs.tmp.dir rather than fully depend on
> hbase-default.xml
>
>
> Key: HBASE-15129
> URL: https://issues.apache.org/jira/browse/HBASE-15129
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15129.patch, HBASE-15129_v2.patch
>
>
> One of our users has observed below error when our cluster upgrades from
> 0.98.12 to 1.1.2:
> {noformat}
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122)
> at org.apache.hadoop.fs.Path.(Path.java:134)
> at org.apache.hadoop.fs.Path.(Path.java:88)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner(HFileOutputFormat2.java:639)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:489)
> {noformat}
> And checking the application code:
> {code}
> private Job createJob(Configuration jobConf) throws IOException {
> String tableName = jobConf.get(TABLE_NAME_KEY);
> Job job = new Job(jobConf);
> Configuration tableConf = HBaseConfiguration.create();
> HTable table = new HTable(tableConf, tableName);
> HFileOutputFormat2.configureIncrementalLoad(job, table);
> return job;
> }
> {code}
> We could see the user has his own hbase-independent job configuration, so our
> current code in HFileOutputFormat2:
> {code}
> Path partitionsPath = new Path(conf.get("hbase.fs.tmp.dir"), "partitions_" +
> UUID.randomUUID());
> {code}
> will return a null string and cause the above exception.
> We propose to add default value in the code rather than depend on
> hbase-default.xml in this JIRA.
> This is an improvement for HBASE-13625
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14490) [RpcServer] reuse request read buffer
[
https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14490:
--
Attachment: gc.png
hits.png
I tried the patch here (sorry it took me so long).
I did two runs, one with, and one without.
Its ycsb load, 50/50, 95% reads, then 80% reads.
The patched run starts at about 13:30.
There is perhaps less GC. This is a macro test so probably not much difference
made by this improvement in the grand scheme of things (Also, what @enis found
where PB CIS is allocating up the wazoo on top of our nice BB from a pool...
probably obscures the benefit too).
I backported the patch to 1.2 to do my run above.
We should update the patch and get it in? (it clashes I'd say w/ commits that
have since gone into master in this area -- I've not checked).
> [RpcServer] reuse request read buffer
> -
>
> Key: HBASE-14490
> URL: https://issues.apache.org/jira/browse/HBASE-14490
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 2.0.0, 1.0.2
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
> Labels: performance
> Fix For: 2.0.0, 1.0.2
>
> Attachments: 14490.hack.to.1.2.patch, ByteBufferPool.java,
> HBASE-14490-v1.patch, HBASE-14490-v10.patch, HBASE-14490-v11.patch,
> HBASE-14490-v12.patch, HBASE-14490-v2.patch, HBASE-14490-v3.patch,
> HBASE-14490-v4.patch, HBASE-14490-v5.patch, HBASE-14490-v6.patch,
> HBASE-14490-v7.patch, HBASE-14490-v8.patch, HBASE-14490-v9.patch, gc.png,
> hits.png, test-v12-patch
>
>
> Reusing buffer to read request.It's not necessary to every request free
> buffer.The idea of optimization is to reduce the times that allocate
> ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}},
> {{processConnectionHeader}} accept a ByteBuffer instead of byte[].They can
> move {{ByteBuffer.position}} correctly when we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a buffer pool in each {{Connection}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15245) Port HBASE-15177 (Reduce garbage created under high load) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15245: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to 0.98. Thanks for taking a look [~enis] > Port HBASE-15177 (Reduce garbage created under high load) to 0.98 > - > > Key: HBASE-15245 > URL: https://issues.apache.org/jira/browse/HBASE-15245 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.18 > > Attachments: HBASE-15245-0.98.patch, HBASE-15245-0.98.patch > > > Evaluate and port back improvements in HBASE-15177 to 0.98 as appropriate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15257) Clean up findbugs warnings on 0.98 branch
Andrew Purtell created HBASE-15257: -- Summary: Clean up findbugs warnings on 0.98 branch Key: HBASE-15257 URL: https://issues.apache.org/jira/browse/HBASE-15257 Project: HBase Issue Type: Task Reporter: Andrew Purtell Priority: Minor Fix For: 0.98.19 Findbugs reports from precommit builds against 0.98 branch will be much more useful if we make a pass over the existing significant warnings and fix them, and add annotations for warnings we expect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15258) Clean up javadoc warnings and errors on 0.98 branch
Andrew Purtell created HBASE-15258: -- Summary: Clean up javadoc warnings and errors on 0.98 branch Key: HBASE-15258 URL: https://issues.apache.org/jira/browse/HBASE-15258 Project: HBase Issue Type: Task Reporter: Andrew Purtell Priority: Minor Fix For: 0.98.19 Clean up javadoc warnings and errors on 0.98 branch so issues reported in precommit build results are more meaningful and actionable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15255) Add pointer to linkedin blog on putting jvm logs on fast disk
[ https://issues.apache.org/jira/browse/HBASE-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143710#comment-15143710 ] Hudson commented on HBASE-15255: FAILURE: Integrated in HBase-Trunk_matrix #702 (See [https://builds.apache.org/job/HBase-Trunk_matrix/702/]) HBASE-15255 Add pointer to linkedin blog on putting jvm logs on fast (stack: rev ab50c7c8c64aa00da14167b01a9e599098afe4bc) * src/main/asciidoc/_chapters/performance.adoc > Add pointer to linkedin blog on putting jvm logs on fast disk > - > > Key: HBASE-15255 > URL: https://issues.apache.org/jira/browse/HBASE-15255 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: stack >Assignee: stack > Attachments: 15255.patch > > > Add pointer to linked in blog: > https://engineering.linkedin.com/blog/2016/02/eliminating-large-jvm-gc-pauses-caused-by-background-io-traffic > IIRC, tsdb says to do similar. > Also add into perf section note on native crc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15245) Port HBASE-15177 (Reduce garbage created under high load) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15245: --- Attachment: HBASE-15245-0.98.patch Findbugs warnings don't apply to this change. 0.98 branch could use a general findbugs cleanup. Filed HBASE-15257. Also a general javadoc cleanup. Filed HBASE-15258 Attached updated patch fixing two checkstyle nits introduced by this patch. Will commit shortly unless objection. > Port HBASE-15177 (Reduce garbage created under high load) to 0.98 > - > > Key: HBASE-15245 > URL: https://issues.apache.org/jira/browse/HBASE-15245 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.18 > > Attachments: HBASE-15245-0.98.patch, HBASE-15245-0.98.patch > > > Evaluate and port back improvements in HBASE-15177 to 0.98 as appropriate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15251) During a cluster restart, Hmaster thinks it is a failover by mistake
[
https://issues.apache.org/jira/browse/HBASE-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143717#comment-15143717
]
Ted Yu commented on HBASE-15251:
Looks good overall.
{code}
447 // Check if there are any regions on these servers
448 failover = false;
{code}
failover is false coming to line 448, there is no need to set to false again.
{code}
465 LOG.debug("Found region" + hri + " out on cluster");
{code}
Leave out a space between region and hri in the output.
{code}
574 if (!fs.getFileStatus(dir).isDirectory()) {
575 LOG.warn(dir + " is not a directory");
{code}
I think the warn() should be debug().
{code}
584 if (files[i].isFile() && files[i].getLen() > 0) {
585 LOG.debug(dir + " has a non-empty file: " + files[i].getPath());
{code}
For file, the above log is inaccurate.
> During a cluster restart, Hmaster thinks it is a failover by mistake
>
>
> Key: HBASE-15251
> URL: https://issues.apache.org/jira/browse/HBASE-15251
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 2.0.0, 0.98.15
>Reporter: Clara Xiong
>Assignee: Clara Xiong
> Attachments: HBASE-15251-master.patch
>
>
> We often need to do cluster restart as part of release for a cluster of >
> 1000 nodes. We have tried our best to get clean shutdown but 50% of the time,
> hmaster still thinks it is a failover. This increases the restart time from 5
> min to 30 min and decreases locality from 99% to 5% since we didn't use a
> locality-aware balancer. We had a bug HBASE-14129 but the fix didn't work.
> After adding more logging and inspecting the logs, we identified two things
> that trigger the failover handling:
> 1. When Hmaster.AssignmentManager detects any dead servers on service
> manager during joinCluster(), it determines this is a failover without
> further check. I added a check whether there is even any region assigned to
> these servers. During a clean restart, the regions are not even assigned.
> 2. When there are some leftover empty folders for log and split directories
> or empty wal files, it is also treated as a failover. I added a check for
> that. Although this can be resolved by manual cleanup, it is still too
> tedious for restarting a large cluster.
> Patch will follow shortly. The fix is tested and used in production now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15251) During a cluster restart, Hmaster thinks it is a failover by mistake
[ https://issues.apache.org/jira/browse/HBASE-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Clara Xiong updated HBASE-15251: Status: Patch Available (was: Open) > During a cluster restart, Hmaster thinks it is a failover by mistake > > > Key: HBASE-15251 > URL: https://issues.apache.org/jira/browse/HBASE-15251 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.98.15, 2.0.0 >Reporter: Clara Xiong >Assignee: Clara Xiong > Attachments: HBASE-15251-master.patch > > > We often need to do cluster restart as part of release for a cluster of > > 1000 nodes. We have tried our best to get clean shutdown but 50% of the time, > hmaster still thinks it is a failover. This increases the restart time from 5 > min to 30 min and decreases locality from 99% to 5% since we didn't use a > balancer. We had a bug HBASE-14129 but the fix didn't work. > After adding more logging and inspecting the logs, we identified two things > that trigger the failover handling: > 1. When Hmaster.AssignmentManager detects any dead servers on service > manager during joinCluster(), it determines this is a failover without > further check. I added a check whether there is even any region assigned to > these servers. During a clean restart, the regions are not even assigned. > 2. When there are some leftover empty folders for log and split directories > or empty wal files, it is also treated as a failover. I added a check for > that. Although this can be resolved by manual cleanup, it is still too > tedious for restarting a large cluster. > Patch will follow shortly. The fix is tested and used in production now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15251) During a cluster restart, Hmaster thinks it is a failover by mistake
[ https://issues.apache.org/jira/browse/HBASE-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Clara Xiong updated HBASE-15251: Description: We often need to do cluster restart as part of release for a cluster of > 1000 nodes. We have tried our best to get clean shutdown but 50% of the time, hmaster still thinks it is a failover. This increases the restart time from 5 min to 30 min and decreases locality from 99% to 5% since we didn't use a locality-aware balancer. We had a bug HBASE-14129 but the fix didn't work. After adding more logging and inspecting the logs, we identified two things that trigger the failover handling: 1. When Hmaster.AssignmentManager detects any dead servers on service manager during joinCluster(), it determines this is a failover without further check. I added a check whether there is even any region assigned to these servers. During a clean restart, the regions are not even assigned. 2. When there are some leftover empty folders for log and split directories or empty wal files, it is also treated as a failover. I added a check for that. Although this can be resolved by manual cleanup, it is still too tedious for restarting a large cluster. Patch will follow shortly. The fix is tested and used in production now. was: We often need to do cluster restart as part of release for a cluster of > 1000 nodes. We have tried our best to get clean shutdown but 50% of the time, hmaster still thinks it is a failover. This increases the restart time from 5 min to 30 min and decreases locality from 99% to 5% since we didn't use a balancer. We had a bug HBASE-14129 but the fix didn't work. After adding more logging and inspecting the logs, we identified two things that trigger the failover handling: 1. When Hmaster.AssignmentManager detects any dead servers on service manager during joinCluster(), it determines this is a failover without further check. I added a check whether there is even any region assigned to these servers. During a clean restart, the regions are not even assigned. 2. When there are some leftover empty folders for log and split directories or empty wal files, it is also treated as a failover. I added a check for that. Although this can be resolved by manual cleanup, it is still too tedious for restarting a large cluster. Patch will follow shortly. The fix is tested and used in production now. > During a cluster restart, Hmaster thinks it is a failover by mistake > > > Key: HBASE-15251 > URL: https://issues.apache.org/jira/browse/HBASE-15251 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.0.0, 0.98.15 >Reporter: Clara Xiong >Assignee: Clara Xiong > Attachments: HBASE-15251-master.patch > > > We often need to do cluster restart as part of release for a cluster of > > 1000 nodes. We have tried our best to get clean shutdown but 50% of the time, > hmaster still thinks it is a failover. This increases the restart time from 5 > min to 30 min and decreases locality from 99% to 5% since we didn't use a > locality-aware balancer. We had a bug HBASE-14129 but the fix didn't work. > After adding more logging and inspecting the logs, we identified two things > that trigger the failover handling: > 1. When Hmaster.AssignmentManager detects any dead servers on service > manager during joinCluster(), it determines this is a failover without > further check. I added a check whether there is even any region assigned to > these servers. During a clean restart, the regions are not even assigned. > 2. When there are some leftover empty folders for log and split directories > or empty wal files, it is also treated as a failover. I added a check for > that. Although this can be resolved by manual cleanup, it is still too > tedious for restarting a large cluster. > Patch will follow shortly. The fix is tested and used in production now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15255) Add pointer to linkedin blog on putting jvm logs on fast disk
stack created HBASE-15255: - Summary: Add pointer to linkedin blog on putting jvm logs on fast disk Key: HBASE-15255 URL: https://issues.apache.org/jira/browse/HBASE-15255 Project: HBase Issue Type: Bug Components: documentation Reporter: stack Assignee: stack Add pointer to linked in blog: https://engineering.linkedin.com/blog/2016/02/eliminating-large-jvm-gc-pauses-caused-by-background-io-traffic IIRC, tsdb says to do similar. Also add into perf section note on native crc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14490) [RpcServer] reuse request read buffer
[
https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14490:
--
Attachment: 14490.hack.to.1.2.patch
Hack to make this patch work on 1.2 so I can perf eval it.
> [RpcServer] reuse request read buffer
> -
>
> Key: HBASE-14490
> URL: https://issues.apache.org/jira/browse/HBASE-14490
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
>Affects Versions: 2.0.0, 1.0.2
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
> Labels: performance
> Fix For: 2.0.0, 1.0.2
>
> Attachments: 14490.hack.to.1.2.patch, ByteBufferPool.java,
> HBASE-14490-v1.patch, HBASE-14490-v10.patch, HBASE-14490-v11.patch,
> HBASE-14490-v12.patch, HBASE-14490-v2.patch, HBASE-14490-v3.patch,
> HBASE-14490-v4.patch, HBASE-14490-v5.patch, HBASE-14490-v6.patch,
> HBASE-14490-v7.patch, HBASE-14490-v8.patch, HBASE-14490-v9.patch,
> test-v12-patch
>
>
> Reusing buffer to read request.It's not necessary to every request free
> buffer.The idea of optimization is to reduce the times that allocate
> ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}},
> {{processConnectionHeader}} accept a ByteBuffer instead of byte[].They can
> move {{ByteBuffer.position}} correctly when we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a buffer pool in each {{Connection}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15243) Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList return SEEK_NEXT_USING_HINT
[ https://issues.apache.org/jira/browse/HBASE-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143557#comment-15143557 ] Preston Koprivica commented on HBASE-15243: --- This might be tangential but I think seeking could be a bit more aggressive on the MUST_PASS_ALL case as well. It appears that the filterKeyValue method simply assigns the internal seekHintFilter to the LAST filter that returned a SEEK_NEXT_USING_HINT [1]. It seems like a more aggressive approach would be to seek to the highest hint (Any previous/lower row won't be accepted by the filter returning that seek). Since the seek filter is simply the last filter that was evaluated, there's no guarantee that we're actually seeking as far as we could. Rather than use a single filter, we could collect all the filters that return that code and then return the maximum seek value of all the filters. [1] https://github.com/apache/hbase/blob/682b8ab8a542a903e5807053282693e3a96bad2d/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java#L271 > Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList > return SEEK_NEXT_USING_HINT > -- > > Key: HBASE-15243 > URL: https://issues.apache.org/jira/browse/HBASE-15243 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15243-v1.txt > > > As Preston Koprivica pointed out at the tail of HBASE-4394, when all filters > in a MUST_PASS_ONE FilterList return a SEEK_USING_NEXT_HINT code, we should > return SEEK_NEXT_USING_HINT from the FilterList to utilize the lowest seek > value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15256) Replication access control should be based on destination table
Gary Helmling created HBASE-15256: - Summary: Replication access control should be based on destination table Key: HBASE-15256 URL: https://issues.apache.org/jira/browse/HBASE-15256 Project: HBase Issue Type: Improvement Components: Replication, security Reporter: Gary Helmling HBASE-12916 added access control for replication sinks, where previously it was missing. However, the access control check is only enforced by checking for a global write permission. This is both less granular than the check could be and less intuitive (access is denied even if the source cell has write permission to the table being replicated). There is obviously more performance overhead from doing more granular checks, but if we only do checks on the distinct set of tables/cfs being written, I think the trade-off might be worth it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15243) Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList return SEEK_NEXT_USING_HINT
[
https://issues.apache.org/jira/browse/HBASE-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143654#comment-15143654
]
Ted Yu commented on HBASE-15243:
Maybe you were thinking of something like this:
http://pastebin.com/55gqwQuB
TestFilterList has to be modified in order to pass:
{code}
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
index 440c9f5..3875841 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
@@ -484,7 +484,7 @@ public class TestFilterList {
Arrays.asList(new Filter [] { filterMinHint, filterMaxHint } ));
filterList.filterKeyValue(null);
assertEquals(0,
CellComparator.COMPARATOR.compare(filterList.getNextCellHint(null),
-minKeyValue));
+maxKeyValue));
filterList = new FilterList(Operator.MUST_PASS_ALL,
Arrays.asList(new Filter [] { filterMaxHint, filterMinHint } ));
@@ -498,7 +498,7 @@ public class TestFilterList {
new Filter [] { filterNoHint, filterMinHint, filterMaxHint } ));
filterList.filterKeyValue(null);
assertEquals(0,
CellComparator.COMPARATOR.compare(filterList.getNextCellHint(null),
-minKeyValue));
+maxKeyValue));
filterList = new FilterList(Operator.MUST_PASS_ALL,
Arrays.asList(new Filter [] { filterNoHint, filterMaxHint } ));
filterList.filterKeyValue(null);
{code}
This is at least an incompatible change, right ?
> Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList
> return SEEK_NEXT_USING_HINT
> --
>
> Key: HBASE-15243
> URL: https://issues.apache.org/jira/browse/HBASE-15243
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15243-v1.txt
>
>
> As Preston Koprivica pointed out at the tail of HBASE-4394, when all filters
> in a MUST_PASS_ONE FilterList return a SEEK_USING_NEXT_HINT code, we should
> return SEEK_NEXT_USING_HINT from the FilterList to utilize the lowest seek
> value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14919) Infrastructure refactoring for In-Memory Flush
[ https://issues.apache.org/jira/browse/HBASE-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143709#comment-15143709 ] Hudson commented on HBASE-14919: FAILURE: Integrated in HBase-Trunk_matrix #702 (See [https://builds.apache.org/job/HBase-Trunk_matrix/702/]) HBASE-14919 Refactoring for in-memory flush and compaction (stack: rev 25dfc112dd76134a9a3ce1f2e88c4075aef76557) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHMobStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlushContext.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreScanner.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MutableSegment.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegmentAdapter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MutableCellSetSegmentScanner.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CellSkipListSet.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMemStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CellSet.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCellSkipListSet.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MutableCellSetSegment.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java > Infrastructure refactoring for In-Memory Flush > -- > > Key: HBASE-14919 > URL: https://issues.apache.org/jira/browse/HBASE-14919 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-14919-V01.patch, HBASE-14919-V01.patch, > HBASE-14919-V02.patch, HBASE-14919-V03.patch, HBASE-14919-V04.patch, > HBASE-14919-V04.patch, HBASE-14919-V05.patch, HBASE-14919-V06.patch, > HBASE-14919-V06.patch, HBASE-14919-V07.patch, HBASE-14919-V08.patch, > HBASE-14919-V09.patch, HBASE-14919-V10.patch, HBASE-14919-V11.patch, > HBASE-14919-V12.patch > > > Refactoring the MemStore hierarchy, introducing segment (StoreSegment) as > first-class citizen and decoupling memstore scanner from the memstore > implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15253) Small bug in CellUtil.matchingRow(Cell, byte[])
[
https://issues.apache.org/jira/browse/HBASE-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143512#comment-15143512
]
Hudson commented on HBASE-15253:
FAILURE: Integrated in HBase-Trunk_matrix #701 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/701/])
HBASE-15253 Small bug in CellUtil.matchingRow(Cell, byte[]) (Ram) (ramkrishna:
rev a975408b7c90e2d545a7a490687cddb717d43807)
* hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java
> Small bug in CellUtil.matchingRow(Cell, byte[])
> ---
>
> Key: HBASE-15253
> URL: https://issues.apache.org/jira/browse/HBASE-15253
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15253.patch
>
>
> {code}
> public static boolean matchingRow(final Cell left, final byte[] buf) {
> if (buf == null) {
> return left.getQualifierLength() == 0;
> }
> return matchingRow(left, buf, 0, buf.length);
> }
> {code}
> The 'if' condition should check for left.getRowLength() == 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15255) Add pointer to linkedin blog on putting jvm logs on fast disk
[ https://issues.apache.org/jira/browse/HBASE-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15255: -- Attachment: 15255.patch What I pushed to master. > Add pointer to linkedin blog on putting jvm logs on fast disk > - > > Key: HBASE-15255 > URL: https://issues.apache.org/jira/browse/HBASE-15255 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: stack >Assignee: stack > Attachments: 15255.patch > > > Add pointer to linked in blog: > https://engineering.linkedin.com/blog/2016/02/eliminating-large-jvm-gc-pauses-caused-by-background-io-traffic > IIRC, tsdb says to do similar. > Also add into perf section note on native crc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15255) Add pointer to linkedin blog on putting jvm logs on fast disk
[ https://issues.apache.org/jira/browse/HBASE-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-15255. --- Resolution: Fixed > Add pointer to linkedin blog on putting jvm logs on fast disk > - > > Key: HBASE-15255 > URL: https://issues.apache.org/jira/browse/HBASE-15255 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: stack >Assignee: stack > Attachments: 15255.patch > > > Add pointer to linked in blog: > https://engineering.linkedin.com/blog/2016/02/eliminating-large-jvm-gc-pauses-caused-by-background-io-traffic > IIRC, tsdb says to do similar. > Also add into perf section note on native crc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
