[jira] [Commented] (HBASE-5030) Some tests do not close the HFile.Reader they use, leaving some file descriptors open
[ https://issues.apache.org/jira/browse/HBASE-5030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13169778#comment-13169778 ] Zhihong Yu commented on HBASE-5030: --- Integrated to TRUNK. Thanks for the patch N. Some tests do not close the HFile.Reader they use, leaving some file descriptors open - Key: HBASE-5030 URL: https://issues.apache.org/jira/browse/HBASE-5030 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Trivial Attachments: 5030.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4336) Convert source tree into maven modules
[ https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13169784#comment-13169784 ] Zhihong Yu commented on HBASE-4336: --- Now src/main/java/org/apache/hadoop/hbase/client is contained in hbase-core. Would it make sense to create hbase-client module ? Convert source tree into maven modules -- Key: HBASE-4336 URL: https://issues.apache.org/jira/browse/HBASE-4336 Project: HBase Issue Type: Task Components: build Reporter: Gary Helmling Priority: Critical Fix For: 0.94.0 When we originally converted the build to maven we had a single core module defined, but later reverted this to a module-less build for the sake of simplicity. It now looks like it's time to re-address this, as we have an actual need for modules to: * provide a trimmed down client library that applications can make use of * more cleanly support building against different versions of Hadoop, in place of some of the reflection machinations currently required * incorporate the secure RPC engine that depends on some secure Hadoop classes I propose we start simply by refactoring into two initial modules: * core - common classes and utilities, and client-side code and interfaces * server - master and region server implementations and supporting code This would also lay the groundwork for incorporating the HBase security features that have been developed. Once the module structure is in place, security-related features could then be incorporated into a third module -- security -- after normal review and approval. The security module could then depend on secure Hadoop, without modifying the dependencies of the rest of the HBase code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5021) Enforce upper bound on timestamp
[ https://issues.apache.org/jira/browse/HBASE-5021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13169817#comment-13169817 ] Zhihong Yu commented on HBASE-5021: --- @Nicolas: You're welcome to comment on HBASE-4605. Enforce upper bound on timestamp Key: HBASE-5021 URL: https://issues.apache.org/jira/browse/HBASE-5021 Project: HBase Issue Type: Improvement Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Priority: Critical Fix For: 0.94.0 Attachments: D849.1.patch We have been getting hit with performance problems on our time-series database due to invalid timestamps being inserted by the timestamp. We are working on adding proper checks to app server, but production performance could be severely impacted with significant recovery time if something slips past. Since timestamps are considered a fundamental part of the HBase schema multiple optimizations use timestamp information, we should allow the option to sanity check the upper bound on the server-side in HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4720) Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server
[ https://issues.apache.org/jira/browse/HBASE-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13169887#comment-13169887 ] Zhihong Yu commented on HBASE-4720: --- I think option 2 is more favorable. Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server Key: HBASE-4720 URL: https://issues.apache.org/jira/browse/HBASE-4720 Project: HBase Issue Type: Improvement Reporter: Daniel Lord I have several large application/HBase clusters where an application node will occasionally need to talk to HBase from a different cluster. In order to help ensure some of my consistency guarantees I have a sentinel table that is updated atomically as users interact with the system. This works quite well for the regular hbase client but the REST client does not implement the checkAndPut and checkAndDelete operations. This exposes the application to some race conditions that have to be worked around. It would be ideal if the same checkAndPut/checkAndDelete operations could be supported by the REST client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168637#comment-13168637 ] Zhihong Yu commented on HBASE-4120: --- In PriorityJobQueue, the call to addLock.unlock() should be wrapped in finally block. isolation and allocation Key: HBASE-4120 URL: https://issues.apache.org/jira/browse/HBASE-4120 Project: HBase Issue Type: New Feature Components: master, regionserver Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0 Reporter: Liu Jia Assignee: Liu Jia Fix For: 0.94.0 Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf, HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch, TablePriority_v12.patch, TablePriority_v12.patch, TablePriority_v15_with_coprocessor.patch, TablePriority_v8.patch, TablePriority_v8.patch, TablePriority_v8_for_trunk.patch, TablePrioriy_v9.patch The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables. When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished. After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group. In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size. Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster. git entry : https://github.com/ICT-Ope/HBase_allocation . We hope our work is helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4993) Performance regression in minicluster creation
[ https://issues.apache.org/jira/browse/HBASE-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168882#comment-13168882 ] Zhihong Yu commented on HBASE-4993: --- testCheckHBaseAvailableClosesConnection hangs without this patch :-( Performance regression in minicluster creation -- Key: HBASE-4993 URL: https://issues.apache.org/jira/browse/HBASE-4993 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Attachments: 4993.patch, 4993.v3.patch Side effect of 4610: the mini cluster needs 4,5 seconds to start -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Delta Encoding of KeyValues (aka prefix compression)
[
https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168935#comment-13168935
]
Zhihong Yu commented on HBASE-4218:
---
There are two files which need to be refreshed:
{code}
1 out of 2 hunks FAILED -- saving rejects to file
src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java.rej
14 out of 14 hunks ignored -- saving rejects to file
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java.rej
{code}
Delta Encoding of KeyValues (aka prefix compression)
-
Key: HBASE-4218
URL: https://issues.apache.org/jira/browse/HBASE-4218
Project: HBase
Issue Type: Improvement
Components: io
Affects Versions: 0.94.0
Reporter: Jacek Migdal
Assignee: Mikhail Bautin
Labels: compression
Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch,
D447.1.patch, D447.2.patch, D447.3.patch, D447.4.patch, D447.5.patch,
D447.6.patch, D447.7.patch, Delta_encoding_with_memstore_TS.patch,
open-source.diff
A compression for keys. Keys are sorted in HFile and they are usually very
similar. Because of that, it is possible to design better compression than
general purpose algorithms,
It is an additional step designed to be used in memory. It aims to save
memory in cache as well as speeding seeks within HFileBlocks. It should
improve performance a lot, if key lengths are larger than value lengths. For
example, it makes a lot of sense to use it when value is a counter.
Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes)
shows that I could achieve decent level of compression:
key compression ratio: 92%
total compression ratio: 85%
LZO on the same data: 85%
LZO after delta encoding: 91%
While having much better performance (20-80% faster decompression ratio than
LZO). Moreover, it should allow far more efficient seeking which should
improve performance a bit.
It seems that a simple compression algorithms are good enough. Most of the
savings are due to prefix compression, int128 encoding, timestamp diffs and
bitfields to avoid duplication. That way, comparisons of compressed data can
be much faster than a byte comparator (thanks to prefix compression and
bitfields).
In order to implement it in HBase two important changes in design will be
needed:
-solidify interface to HFileBlock / HFileReader Scanner to provide seeking
and iterating; access to uncompressed buffer in HFileBlock will have bad
performance
-extend comparators to support comparison assuming that N first bytes are
equal (or some fields are equal)
Link to a discussion about something similar:
http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windowssubj=Re+prefix+compression
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4616) Update hregion encoded name to reduce logic and prevent region collisions in META
[
https://issues.apache.org/jira/browse/HBASE-4616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13168953#comment-13168953
]
Zhihong Yu commented on HBASE-4616:
---
There is error in the name of the latest patch - it should be 4616.
TestOfflineMetaRebuildBase failed with:
{code}
Caused by: java.lang.RuntimeException: Master not initialized after 200 seconds
at
org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:206)
at
org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:420)
at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:196)
{code}
Update hregion encoded name to reduce logic and prevent region collisions in
META
-
Key: HBASE-4616
URL: https://issues.apache.org/jira/browse/HBASE-4616
Project: HBase
Issue Type: Umbrella
Reporter: Alex Newman
Assignee: Alex Newman
Attachments: HBASE-2616-v2.patch, HBASE-4616.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13169075#comment-13169075
]
Zhihong Yu commented on HBASE-4605:
---
TestHTableDescriptor misses category.
Other than that I vote +1 on latest patch.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: 4605.v7, constraint_as_cp.txt, java_Constraint_v2.patch,
java_HBASE-4605_v1.patch, java_HBASE-4605_v2.patch, java_HBASE-4605_v3.patch,
java_HBASE-4605_v5.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4970) Add a parameter to change keepAliveTime of Htable thread pool.
[ https://issues.apache.org/jira/browse/HBASE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164580#comment-13164580 ] Zhihong Yu commented on HBASE-4970: --- Patch v2 is a backport and doesn't change keepAliveTime. I feel we should address the needs of HTable users. I am fine with the backport - we may want to modify the title of this JIRA accordingly. Add a parameter to change keepAliveTime of Htable thread pool. --- Key: HBASE-4970 URL: https://issues.apache.org/jira/browse/HBASE-4970 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.90.4 Reporter: gaojinchao Assignee: gaojinchao Priority: Trivial Fix For: 0.90.5 Attachments: HBASE-4970_Branch90.patch, HBASE-4970_Branch90_V1_trial.patch In my cluster, I changed keepAliveTime from 60 s to 3600 s. Increasing RES is slowed down. Why increasing keepAliveTime of HBase thread pool is slowing down our problem occurance [RES value increase]? You can go through the source of sun.nio.ch.Util. Every thread hold 3 softreference of direct buffer(mustangsrc) for reusage. The code names the 3 softreferences buffercache. If the buffer was all occupied or none was suitable in size, and new request comes, new direct buffer is allocated. After the service, the bigger one replaces the smaller one in buffercache. The replaced buffer is released. So I think we can add a parameter to change keepAliveTime of Htable thread pool. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4880) Region is on service before completing openRegionHanlder, may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164605#comment-13164605
]
Zhihong Yu commented on HBASE-4880:
---
The three test failures would be fixed by addendum to HBASE-4927.
Region is on service before completing openRegionHanlder, may cause data loss
-
Key: HBASE-4880
URL: https://issues.apache.org/jira/browse/HBASE-4880
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: hbase-4880.patch, hbase-4880v2.patch, hbase-4880v3.patch
OpenRegionHandler in regionserver is processed as the following steps:
{code}
1.openregion()(Through it, closed = false, closing = false)
2.addToOnlineRegions(region)
3.update .meta. table
4.update ZK's node state to RS_ZK_REGION_OPEND
{code}
We can find that region is on service before Step 4.
It means client could put data to this region after step 3.
What will happen if step 4 is failed processing?
It will execute OpenRegionHandler#cleanupFailedOpen which will do closing
region, and master assign this region to another regionserver.
If closing region is failed, the data which is put between step 3 and step 4
may loss, because the region has been opend on another regionserver and be
put new data. Therefore, it may not be recoverd through replayRecoveredEdit()
because the edit's LogSeqId is smaller than current region SeqId.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3857) Change the HFile Format
[ https://issues.apache.org/jira/browse/HBASE-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164618#comment-13164618 ] Zhihong Yu commented on HBASE-3857: --- @J-D: Nice catch. We should open another JIRA to deal with queue sizes. Change the HFile Format --- Key: HBASE-3857 URL: https://issues.apache.org/jira/browse/HBASE-3857 Project: HBase Issue Type: New Feature Affects Versions: 0.90.4 Reporter: Liyin Tang Assignee: Mikhail Bautin Attachments: 0001-Adding-release-notes-for-HBASE-3857.patch, 0001-Fix-TestHFileBlock.testBlockHeapSize.patch, 0001-review_hfile-v2-r1144693_2011-07-15_11_14_44.patch, 0001-review_hfile-v2-r1147350_2011-07-26_11_55_59.patch, 0001-review_hfile-v2-r1152122-2011_08_01_03_18_00.patch, 0001-review_hfile-v2-r1153300-git-1152532-2011_08_02_19_4.patch, 0001-review_hfile-v2-r1153300-git-1152532-2011_08_03_12_4.patch, hfile_format_v2_design_draft_0.1.pdf, hfile_format_v2_design_draft_0.3.pdf, hfile_format_v2_design_draft_0.4.odt In order to support HBASE-3763 and HBASE-3856, we need to change the format of the HFile. The new format proposal is attached here. Thanks for Mikhail Bautin for the documentation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4880) Region is on service before openRegionHandler completes, may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164777#comment-13164777
]
Zhihong Yu commented on HBASE-4880:
---
Latest patch passes all tests.
+1.
Region is on service before openRegionHandler completes, may cause data loss
Key: HBASE-4880
URL: https://issues.apache.org/jira/browse/HBASE-4880
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: 4880.txt, hbase-4880.patch, hbase-4880v2.patch,
hbase-4880v3.patch
OpenRegionHandler in regionserver is processed as the following steps:
{code}
1.openregion()(Through it, closed = false, closing = false)
2.addToOnlineRegions(region)
3.update .meta. table
4.update ZK's node state to RS_ZK_REGION_OPEND
{code}
We can find that region is on service before Step 4.
It means client could put data to this region after step 3.
What will happen if step 4 is failed processing?
It will execute OpenRegionHandler#cleanupFailedOpen which will do closing
region, and master assign this region to another regionserver.
If closing region is failed, the data which is put between step 3 and step 4
may loss, because the region has been opend on another regionserver and be
put new data. Therefore, it may not be recoverd through replayRecoveredEdit()
because the edit's LogSeqId is smaller than current region SeqId.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4610) Port HBASE-3380 (Master failover can split logs of live servers) to 92/trunk (definitely bring in config params, decide if we need to do more to fix the bug)
[ https://issues.apache.org/jira/browse/HBASE-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164785#comment-13164785 ] Zhihong Yu commented on HBASE-4610: --- Test suite passes. Will commit later today if no objections. Port HBASE-3380 (Master failover can split logs of live servers) to 92/trunk (definitely bring in config params, decide if we need to do more to fix the bug) - Key: HBASE-4610 URL: https://issues.apache.org/jira/browse/HBASE-4610 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0, 0.94.0 Reporter: Jonathan Gray Assignee: Jonathan Gray Fix For: 0.92.1 Attachments: 4610.txt Over in HBASE-3380 we were having some TestMasterFailover flakiness. We added some more config parameters to better control the master startup loop where it waits for RS to heartbeat in. We had thought at the time that 92 would have a different solution but it is still relying on heartbeats to learn about RSs. For now, we should definitely bring these config params into 92/trunk. Otherwise this is an incompatible regression and adding these will also make things like what was just reported over in HBASE-4603 trivial to fix in an optimal way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4956) Control direct memory buffer consumption by HBaseClient
[ https://issues.apache.org/jira/browse/HBASE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164798#comment-13164798 ] Zhihong Yu commented on HBASE-4956: --- Since the proposal involves asynchronous communication, we should devise new API which can be used to validate the reduction in use of direct memory buffer. Control direct memory buffer consumption by HBaseClient --- Key: HBASE-4956 URL: https://issues.apache.org/jira/browse/HBASE-4956 Project: HBase Issue Type: New Feature Reporter: Ted Yu As Jonathan explained here https://groups.google.com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357?pli=1 , standard hbase client inadvertently consumes large amount of direct memory. We should consider using netty for NIO-related tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4610) Port HBASE-3380 (Master failover can split logs of live servers) to 92/trunk (definitely bring in config params, decide if we need to do more to fix the bug)
[ https://issues.apache.org/jira/browse/HBASE-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164839#comment-13164839 ] Zhihong Yu commented on HBASE-4610: --- Thanks for the review Jonathan. The first break statement should be removed. I ran TestMasterFailover on MacBook and didn't see failure. I think this should go to 0.92.0 Port HBASE-3380 (Master failover can split logs of live servers) to 92/trunk (definitely bring in config params, decide if we need to do more to fix the bug) - Key: HBASE-4610 URL: https://issues.apache.org/jira/browse/HBASE-4610 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0, 0.94.0 Reporter: Jonathan Gray Assignee: Jonathan Gray Fix For: 0.92.1 Attachments: 4610.txt Over in HBASE-3380 we were having some TestMasterFailover flakiness. We added some more config parameters to better control the master startup loop where it waits for RS to heartbeat in. We had thought at the time that 92 would have a different solution but it is still relying on heartbeats to learn about RSs. For now, we should definitely bring these config params into 92/trunk. Otherwise this is an incompatible regression and adding these will also make things like what was just reported over in HBASE-4603 trivial to fix in an optimal way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4880) Region is on service before completing openRegionHanlder, may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163943#comment-13163943
]
Zhihong Yu commented on HBASE-4880:
---
TestAdmin#testForceSplit would fail if I comment out the following line in
HRegionServer.postOpenDeployTasks():
{code}
-addToOnlineRegions(r);
+// addToOnlineRegions(r);
{code}
This is the error:
{code}
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: NotServingRegionException: 1 time, servers with issues:
lm-sjn-xx:65442,
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1641)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:892)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:750)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:725)
at
org.apache.hadoop.hbase.client.TestAdmin.splitTest(TestAdmin.java:805)
at
org.apache.hadoop.hbase.client.TestAdmin.testForceSplit(TestAdmin.java:108)
{code}
Region is on service before completing openRegionHanlder, may cause data loss
-
Key: HBASE-4880
URL: https://issues.apache.org/jira/browse/HBASE-4880
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: hbase-4880.patch
OpenRegionHandler in regionserver is processed as the following steps:
{code}
1.openregion()(Through it, closed = false, closing = false)
2.addToOnlineRegions(region)
3.update .meta. table
4.update ZK's node state to RS_ZK_REGION_OPEND
{code}
We can find that region is on service before Step 4.
It means client could put data to this region after step 3.
What will happen if step 4 is failed processing?
It will execute OpenRegionHandler#cleanupFailedOpen which will do closing
region, and master assign this region to another regionserver.
If closing region is failed, the data which is put between step 3 and step 4
may loss, because the region has been opend on another regionserver and be
put new data. Therefore, it may not be recoverd through replayRecoveredEdit()
because the edit's LogSeqId is smaller than current region SeqId.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13163944#comment-13163944
]
Zhihong Yu commented on HBASE-4605:
---
w.r.t. atomicity, it is outside the initial scope of this JIRA.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: 4605.v7, constraint_as_cp.txt, java_Constraint_v2.patch,
java_HBASE-4605_v1.patch, java_HBASE-4605_v2.patch, java_HBASE-4605_v3.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4880) Region is on service before completing openRegionHanlder, may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164122#comment-13164122
]
Zhihong Yu commented on HBASE-4880:
---
Patch v2 is cleaner.
Minor comment:
The change in RegionServerServices.java shouldn't put explanation for
parameters on a separate line - parameter name and explanation should be on the
same line.
Good job Chunhui.
Region is on service before completing openRegionHanlder, may cause data loss
-
Key: HBASE-4880
URL: https://issues.apache.org/jira/browse/HBASE-4880
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: hbase-4880.patch, hbase-4880v2.patch
OpenRegionHandler in regionserver is processed as the following steps:
{code}
1.openregion()(Through it, closed = false, closing = false)
2.addToOnlineRegions(region)
3.update .meta. table
4.update ZK's node state to RS_ZK_REGION_OPEND
{code}
We can find that region is on service before Step 4.
It means client could put data to this region after step 3.
What will happen if step 4 is failed processing?
It will execute OpenRegionHandler#cleanupFailedOpen which will do closing
region, and master assign this region to another regionserver.
If closing region is failed, the data which is put between step 3 and step 4
may loss, because the region has been opend on another regionserver and be
put new data. Therefore, it may not be recoverd through replayRecoveredEdit()
because the edit's LogSeqId is smaller than current region SeqId.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4970) Add a parameter to change keepAliveTime of Htable thread pool.
[ https://issues.apache.org/jira/browse/HBASE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13164123#comment-13164123 ] Zhihong Yu commented on HBASE-4970: --- @Jinchao: Can you upload a patch ? Thanks Add a parameter to change keepAliveTime of Htable thread pool. --- Key: HBASE-4970 URL: https://issues.apache.org/jira/browse/HBASE-4970 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.90.4 Reporter: gaojinchao Assignee: gaojinchao Priority: Trivial Fix For: 0.90.5 In my cluster, I changed keepAliveTime from 60 s to 3600 s. Increasing RES is slowed down. Why increasing keepAliveTime of HBase thread pool is slowing down our problem occurance [RES value increase]? You can go through the source of sun.nio.ch.Util. Every thread hold 3 softreference of direct buffer(mustangsrc) for reusage. The code names the 3 softreferences buffercache. If the buffer was all occupied or none was suitable in size, and new request comes, new direct buffer is allocated. After the service, the bigger one replaces the smaller one in buffercache. The replaced buffer is released. So I think we can add a parameter to change keepAliveTime of Htable thread pool. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
