[jira] [Commented] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072700#comment-13072700 ] Liu Jia commented on HBASE-4120: @Jeff Thanks Jeff, I think to optimize the customer facing tables and control the resource of batch processing tables it's a good use case for priority. In Taobao Company, actually the group is used more often than table priority. Maybe without test or competition on resource, the table priority works quiet. But the group has a user portal and friendly with the users who not very familiar with HBase. The department which holds a shared HBase cluster called Taobao data platform, and many other departments may not very familiar with HBase want to use HBase. Most of them would like to test the performance (mostly is the throughput per region server) with their data and methods first. So the isolation part is very useful for them, I think if HBase wants to be a basic component of data center like Hadoop, a convenient and flexible way to isolate different projects is important. Jeff, is the mentioned part of the complicate of HBase significantly related with the Portal and too many JSP pages? What if add a shell tool to replace them? Because the actually implementation of group is just depends on region assignment and movement. isolation and allocation Key: HBASE-4120 URL: https://issues.apache.org/jira/browse/HBASE-4120 Project: HBase Issue Type: New Feature Components: master, regionserver Affects Versions: 0.90.2 Reporter: Liu Jia Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf, HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables. When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished. After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group. In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size. Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster. git entry : https://github.com/ICT-Ope/HBase_allocation . We hope our work is helpful. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072701#comment-13072701 ] Liu Jia commented on HBASE-4120: @stack Hi stack, when design the table priority part, we thought a good design may need to meet the following four requirements 1, be efficient, don't affect the system throughput. 2, when system is not busy, tables with different priorities no matter high or low can take full use of the system resources. 3, when system is busy, high priority table costs more resource than low. 4, when system is busy, low priority table will be functioning normal. Before carrying out this work, we actually thought about managing the block cache or memstore to achieve the goals, but this is more complex than managing the RPC queue and meanwhile it’s not a directly approach to solve this problem. Keep it simple is the principal that we must follow. Due to the LRU cache replacement strategy of block cache, the more RPC requests processed the more block cache size will be occupied, so we make this implementation decision. Stack, would you please send me the issue number which is about the previous discussion of this function. Other's opinions can help us find defective points and neglected problem in our design. isolation and allocation Key: HBASE-4120 URL: https://issues.apache.org/jira/browse/HBASE-4120 Project: HBase Issue Type: New Feature Components: master, regionserver Affects Versions: 0.90.2 Reporter: Liu Jia Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf, HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables. When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished. After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group. In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size. Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster. git entry : https://github.com/ICT-Ope/HBase_allocation . We hope our work is helpful. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072702#comment-13072702 ] Liu Jia commented on HBASE-4120: @Todd Hi Todd, I think to provide HBase as a service may face the same problem of isolation and prioritization, this part is mentioned by Jeffrey Dean in one of his ppt. but just come up as the following BigTable as a service ? – Interesting issues of resource fairness, performance isolation, prioritization, etc. across different clients But without any further discussion, any more detailed information? The Same part in BigTable will be very interesting and valuable for to refine our designing. May be our group function could integrate with the user management part of HBase in the future. isolation and allocation Key: HBASE-4120 URL: https://issues.apache.org/jira/browse/HBASE-4120 Project: HBase Issue Type: New Feature Components: master, regionserver Affects Versions: 0.90.2 Reporter: Liu Jia Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf, HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables. When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished. After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group. In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size. Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster. git entry : https://github.com/ICT-Ope/HBase_allocation . We hope our work is helpful. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4144) RS does not abort if the initialization of RS fails
[
https://issues.apache.org/jira/browse/HBASE-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-4144:
--
Summary: RS does not abort if the initialization of RS fails (was: RS
doesnot abort if the initialization of RS fails)
RS does not abort if the initialization of RS fails
Key: HBASE-4144
URL: https://issues.apache.org/jira/browse/HBASE-4144
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.3
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Attachments: HBASE-4144_0.90.patch, HBASE-4144_trunk.patch
If any exception occurs while initialization of RS the RS doesnot get
aborted whereas only the RPC server gets stopped.
private void preRegistrationInitialization()
throws IOException, InterruptedException {
try {
initializeZooKeeper();
initializeThreads();
int nbBlocks = conf.getInt(hbase.regionserver.nbreservationblocks,4);
for (int i = 0; i nbBlocks; i++) {
reservedSpace.add(new
byte[HConstants.DEFAULT_SIZE_RESERVATION_BLOCK]);
}
} catch (Throwable t) {
// Call stop if error or process will stick around for ever since
server
// puts up non-daemon threads.
LOG.error(Stopping HRS because failed initialize, t);
this.rpcServer.stop();
}
}
So if any exception occurs while initilization the RPC server gets stopped
but RS process is still running. But the log says stopping HRegionServer.
So in the below code the catch() block will be executed when the RPCServer
stop fails?
In all other cases it doesnt handle any initialization failure.
try {
// Do pre-registration initializations; zookeeper, lease threads,tc.
preRegistrationInitialization();
} catch (Exception e) {
abort(Fatal exception during initialization, e);
}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4144) RS does not abort if the initialization of RS fails
[
https://issues.apache.org/jira/browse/HBASE-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072708#comment-13072708
]
Ted Yu commented on HBASE-4144:
---
Applied to branch and TRUNK.
Thanks for the patch Ramkrishna.
RS does not abort if the initialization of RS fails
Key: HBASE-4144
URL: https://issues.apache.org/jira/browse/HBASE-4144
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.3
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Attachments: HBASE-4144_0.90.patch, HBASE-4144_trunk.patch
If any exception occurs while initialization of RS the RS doesnot get
aborted whereas only the RPC server gets stopped.
private void preRegistrationInitialization()
throws IOException, InterruptedException {
try {
initializeZooKeeper();
initializeThreads();
int nbBlocks = conf.getInt(hbase.regionserver.nbreservationblocks,4);
for (int i = 0; i nbBlocks; i++) {
reservedSpace.add(new
byte[HConstants.DEFAULT_SIZE_RESERVATION_BLOCK]);
}
} catch (Throwable t) {
// Call stop if error or process will stick around for ever since
server
// puts up non-daemon threads.
LOG.error(Stopping HRS because failed initialize, t);
this.rpcServer.stop();
}
}
So if any exception occurs while initilization the RPC server gets stopped
but RS process is still running. But the log says stopping HRegionServer.
So in the below code the catch() block will be executed when the RPCServer
stop fails?
In all other cases it doesnt handle any initialization failure.
try {
// Do pre-registration initializations; zookeeper, lease threads,tc.
preRegistrationInitialization();
} catch (Exception e) {
abort(Fatal exception during initialization, e);
}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4138) If zookeeper.znode.parent is not specifed explicitly in Client code then HTable object loops continuously waiting for the root region by using /hbase as the base node.
[ https://issues.apache.org/jira/browse/HBASE-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072711#comment-13072711 ] Ted Yu commented on HBASE-4138: --- I don't think failure of TestSplitLogManager was related to this JIRA either. TestSplitLogManager hung in build 2061 on Jenkins. If zookeeper.znode.parent is not specifed explicitly in Client code then HTable object loops continuously waiting for the root region by using /hbase as the base node. --- Key: HBASE-4138 URL: https://issues.apache.org/jira/browse/HBASE-4138 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.90.3 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.92.0 Attachments: HBASE-4138_trunk_1.patch, HBASE-4138_trunk_2.patch, HBASE-4138_trunk_3.patch Change the zookeeper.znode.parent property (default is /hbase). Now do not specify this change in the client code. Use the HTable Object. The HTable is not able to find the root region and keeps continuously looping. Find the stack trace: Object.wait(long) line: not available [native method] RootRegionTracker(ZooKeeperNodeTracker).blockUntilAvailable(long) line: 122 RootRegionTracker.waitRootRegionLocation(long) line: 73 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[], boolean) line: 578 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[]) line: 558 HConnectionManager$HConnectionImplementation.locateRegionInMeta(byte[], byte[], byte[], boolean, Object) line: 687 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[], boolean) line: 589 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[]) line: 558 HConnectionManager$HConnectionImplementation.locateRegionInMeta(byte[], byte[], byte[], boolean, Object) line: 687 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[], boolean) line: 593 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[]) line: 558 HTable.init(Configuration, byte[]) line: 171 HTable.init(Configuration, String) line: 145 HBaseTest.test() line: 45 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072716#comment-13072716
]
Li Pi commented on HBASE-4027:
--
The bug actually broke an existing test case. I'm not sure why I didn't notice
- I did after trying to rebuild it this morning.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4144) RS does not abort if the initialization of RS fails
[
https://issues.apache.org/jira/browse/HBASE-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072744#comment-13072744
]
Hudson commented on HBASE-4144:
---
Integrated in HBase-TRUNK #2062 (See
[https://builds.apache.org/job/HBase-TRUNK/2062/])
HBASE-4144 RS does not abort if the initialization of RS fails
(ramkrishna.s.vasudevan)
tedyu :
Files :
* /hbase/trunk/CHANGES.txt
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
RS does not abort if the initialization of RS fails
Key: HBASE-4144
URL: https://issues.apache.org/jira/browse/HBASE-4144
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.3
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Attachments: HBASE-4144_0.90.patch, HBASE-4144_trunk.patch
If any exception occurs while initialization of RS the RS doesnot get
aborted whereas only the RPC server gets stopped.
private void preRegistrationInitialization()
throws IOException, InterruptedException {
try {
initializeZooKeeper();
initializeThreads();
int nbBlocks = conf.getInt(hbase.regionserver.nbreservationblocks,4);
for (int i = 0; i nbBlocks; i++) {
reservedSpace.add(new
byte[HConstants.DEFAULT_SIZE_RESERVATION_BLOCK]);
}
} catch (Throwable t) {
// Call stop if error or process will stick around for ever since
server
// puts up non-daemon threads.
LOG.error(Stopping HRS because failed initialize, t);
this.rpcServer.stop();
}
}
So if any exception occurs while initilization the RPC server gets stopped
but RS process is still running. But the log says stopping HRegionServer.
So in the below code the catch() block will be executed when the RPCServer
stop fails?
In all other cases it doesnt handle any initialization failure.
try {
// Do pre-registration initializations; zookeeper, lease threads,tc.
preRegistrationInitialization();
} catch (Exception e) {
abort(Fatal exception during initialization, e);
}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss
[
https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072769#comment-13072769
]
ramkrishna.s.vasudevan commented on HBASE-3065:
---
Reason for TestSplitLogManager failures
===
Pls find the analysis
- The Recoverable zookeeper encodes the node name while creating.(This is
already pointed out by Stack).
- The RecoverableZookeeper while writing the data adds some metadata to it.
{noformat}
byte[] newData = appendMetaData(data);
{noformat}
While we executing some of the testcases
SplitLogManager.getDataSetWatchSuccess() gets invoked.
Here we do some state comparions like
{noformat}
if (TaskState.TASK_UNASSIGNED.equals(data)) {
LOG.debug(task not yet acquired + path + ver = + version);
handleUnassignedTask(path);
} else if (TaskState.TASK_OWNED.equals(data)) {
registerHeartbeat(path, version,
TaskState.TASK_OWNED.getWriterName(data));
} else if (TaskState.TASK_RESIGNED.equals(data)) {
LOG.info(task + path + entered state + new String(data));
resubmit(path, true);
}
{noformat}
Here the data variable is with metadata appended while writing the data whereas
the TaskState is without metadata.
So any comparison that we make fails.
Also one more observation is 'testOrphanTaskAcquisition()' testcase needs some
wait mechanism before proceeding.
Because the GetDataAsyncCallback call is asynchronous.
In RecoverableZooKeeper the javadoc itself says creating a node should be
handled carefully.
I have still not completely covered all the failures but this is the basic
reason. Even the test case 'testTaskDone'()' hanging is due to the same
problem I feel.
Am not fully aware of the splitlog feature with zookeeper will try to provide
an addendum to this.
Retry all 'retryable' zk operations; e.g. connection loss
-
Key: HBASE-3065
URL: https://issues.apache.org/jira/browse/HBASE-3065
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: Liyin Tang
Priority: Critical
Fix For: 0.92.0
Attachments: 3065-v3.txt, 3065-v4.txt, HBase-3065[r1088475]_1.patch,
hbase3065_2.patch
The 'new' master refactored our zk code tidying up all zk accesses and
coralling them behind nice zk utility classes. One improvement was letting
out all KeeperExceptions letting the client deal. Thats good generally
because in old days, we'd suppress important state zk changes in state. But
there is at least one case the new zk utility could handle for the
application and thats the class of retryable KeeperExceptions. The one that
comes to mind is conection loss. On connection loss we should retry the
just-failed operation. Usually the retry will just work. At worse, on
reconnect, we'll pick up the expired session event.
Adding in this change shouldn't be too bad given the refactor of zk corralled
all zk access into one or two classes only.
One thing to consider though is how much we should retry. We could retry on
a timer or we could retry for ever as long as the Stoppable interface is
passed so if another thread has stopped or aborted the hosting service, we'll
notice and give up trying. Doing the latter is probably better than some
kinda timeout.
HBASE-3062 adds a timed retry on the first zk operation. This issue is about
generalizing what is over there across all zk access.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss
[ https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072770#comment-13072770 ] ramkrishna.s.vasudevan commented on HBASE-3065: --- The above analysis is for the testcase 'testOrphanTaskAcquisition()' Retry all 'retryable' zk operations; e.g. connection loss - Key: HBASE-3065 URL: https://issues.apache.org/jira/browse/HBASE-3065 Project: HBase Issue Type: Bug Reporter: stack Assignee: Liyin Tang Priority: Critical Fix For: 0.92.0 Attachments: 3065-v3.txt, 3065-v4.txt, HBase-3065[r1088475]_1.patch, hbase3065_2.patch The 'new' master refactored our zk code tidying up all zk accesses and coralling them behind nice zk utility classes. One improvement was letting out all KeeperExceptions letting the client deal. Thats good generally because in old days, we'd suppress important state zk changes in state. But there is at least one case the new zk utility could handle for the application and thats the class of retryable KeeperExceptions. The one that comes to mind is conection loss. On connection loss we should retry the just-failed operation. Usually the retry will just work. At worse, on reconnect, we'll pick up the expired session event. Adding in this change shouldn't be too bad given the refactor of zk corralled all zk access into one or two classes only. One thing to consider though is how much we should retry. We could retry on a timer or we could retry for ever as long as the Stoppable interface is passed so if another thread has stopped or aborted the hosting service, we'll notice and give up trying. Doing the latter is probably better than some kinda timeout. HBASE-3062 adds a timed retry on the first zk operation. This issue is about generalizing what is over there across all zk access. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4143) HTable.doPut(List) should check the writebuffer length every so often
[
https://issues.apache.org/jira/browse/HBASE-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072771#comment-13072771
]
Doug Meil commented on HBASE-4143:
--
re: This effectively disables the ability to do batching.
There is already a client method called 'batch'. I think that should be
encouraged to be the preferred batch method if callers want a do exactly what
I say approach. Otherwise, put(Put) and put(List) should obey the writeBuffer
rules. I'm cool with the patch though.
HTable.doPut(List) should check the writebuffer length every so often
-
Key: HBASE-4143
URL: https://issues.apache.org/jira/browse/HBASE-4143
Project: HBase
Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
Attachments: HBASE-4143_update.patch, client_HBASE_4143.patch
This came up on a dist-list conversation between Andy P., Ted Yu, and myself.
Andy noted that extremely large lists passed into put(List) can cause
issues. Ted suggested that having doPut check the write-buffer length every
so often (5-10 records?) so the flush doesn't happen only at the end, and I
think that's good idea.
public void put(final ListPut puts) throws IOException {
doPut(puts);
}
private void doPut(final ListPut puts) throws IOException {
for (Put put : puts) {
validatePut(put);
writeBuffer.add(put);
currentWriteBufferSize += put.heapSize();
}
if (autoFlush || currentWriteBufferSize writeBufferSize) {
flushCommits();
}
}
Once this change is made, remove the comment in HBASE-4142 about large lists
being a performance problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4147) StoreFile query usage report
StoreFile query usage report Key: HBASE-4147 URL: https://issues.apache.org/jira/browse/HBASE-4147 Project: HBase Issue Type: Improvement Reporter: Doug Meil Priority: Minor Detailed information on what HBase is doing in terms of reads is hard to come by. What would be useful is to have a periodic StoreFile query report. Specifically, this could run on a configured interval (e.g., every 30 seconds, 60 seconds) and dump the output to the log files. This would have all StoreFiles accessed during the reporting period (and with the Path we would also know region, CF, and table), # of times the StoreFile was accessed, the size of the StoreFile, and the total time (ms) spent processing that StoreFile. Even this level of summary would be useful to detect a which tables CFs are being accessed the most, and including the StoreFile would provide insight into relative uncompaction (i.e., lots of StoreFiles). I think the log-output, as opposed to UI, is an important facet with this. I'm assuming that users will slice and dice this data on their own so I think we should skip any kind of admin view for now (i.e., new JSPs, new APIs to expose this data). Just getting this to log-file would be a big improvement. Will this have a non-zero performance impact? Yes. Hopefully small, but yes it will. However, flying a plane without any instrumentation isn't fun. :-) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4089) blockCache contents report
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4089: - Description: Summarized block-cache report for a RegionServer would be helpful. For example ... table1 cf1 100 blocks, totalBytes=y, averageTimeInCache= hours cf2 200 blocks, totalBytes=z, averageTimeInCache= hours table2 cf1 75 blocks, totalBytes=y, averageTimeInCache= hours cf2 150 blocks, totalBytes=z, averageTimeInCache= hours ... Etc. The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important. No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective. Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file. was: A UI that would display a block-cache report for a RegionServer would be helpful. For example ... table1 cf1 100 blocks, totalBytes=y, averageTimeInCache= hours cf2 200 blocks, totalBytes=z, averageTimeInCache= hours table2 cf1 75 blocks, totalBytes=y, averageTimeInCache= hours cf2 150 blocks, totalBytes=z, averageTimeInCache= hours ... Etc. The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important. No such interface exists in HRegionInterface, so this is not just a UI request also an API change. But I think it would be helpful from an operational perspective. blockCache contents report -- Key: HBASE-4089 URL: https://issues.apache.org/jira/browse/HBASE-4089 Project: HBase Issue Type: New Feature Reporter: Doug Meil Summarized block-cache report for a RegionServer would be helpful. For example ... table1 cf1 100 blocks, totalBytes=y, averageTimeInCache= hours cf2 200 blocks, totalBytes=z, averageTimeInCache= hours table2 cf1 75 blocks, totalBytes=y, averageTimeInCache= hours cf2 150 blocks, totalBytes=z, averageTimeInCache= hours ... Etc. The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important. No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective. Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss
[ https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072779#comment-13072779 ] ramkrishna.s.vasudevan commented on HBASE-3065: --- I will upload the patch. I got the exact fix Retry all 'retryable' zk operations; e.g. connection loss - Key: HBASE-3065 URL: https://issues.apache.org/jira/browse/HBASE-3065 Project: HBase Issue Type: Bug Reporter: stack Assignee: Liyin Tang Priority: Critical Fix For: 0.92.0 Attachments: 3065-v3.txt, 3065-v4.txt, HBase-3065[r1088475]_1.patch, hbase3065_2.patch The 'new' master refactored our zk code tidying up all zk accesses and coralling them behind nice zk utility classes. One improvement was letting out all KeeperExceptions letting the client deal. Thats good generally because in old days, we'd suppress important state zk changes in state. But there is at least one case the new zk utility could handle for the application and thats the class of retryable KeeperExceptions. The one that comes to mind is conection loss. On connection loss we should retry the just-failed operation. Usually the retry will just work. At worse, on reconnect, we'll pick up the expired session event. Adding in this change shouldn't be too bad given the refactor of zk corralled all zk access into one or two classes only. One thing to consider though is how much we should retry. We could retry on a timer or we could retry for ever as long as the Stoppable interface is passed so if another thread has stopped or aborted the hosting service, we'll notice and give up trying. Doing the latter is probably better than some kinda timeout. HBASE-3062 adds a timed retry on the first zk operation. This issue is about generalizing what is over there across all zk access. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss
[ https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-3065: -- Attachment: HBASE-3065-addendum.patch Retry all 'retryable' zk operations; e.g. connection loss - Key: HBASE-3065 URL: https://issues.apache.org/jira/browse/HBASE-3065 Project: HBase Issue Type: Bug Reporter: stack Assignee: Liyin Tang Priority: Critical Fix For: 0.92.0 Attachments: 3065-v3.txt, 3065-v4.txt, HBASE-3065-addendum.patch, HBase-3065[r1088475]_1.patch, hbase3065_2.patch The 'new' master refactored our zk code tidying up all zk accesses and coralling them behind nice zk utility classes. One improvement was letting out all KeeperExceptions letting the client deal. Thats good generally because in old days, we'd suppress important state zk changes in state. But there is at least one case the new zk utility could handle for the application and thats the class of retryable KeeperExceptions. The one that comes to mind is conection loss. On connection loss we should retry the just-failed operation. Usually the retry will just work. At worse, on reconnect, we'll pick up the expired session event. Adding in this change shouldn't be too bad given the refactor of zk corralled all zk access into one or two classes only. One thing to consider though is how much we should retry. We could retry on a timer or we could retry for ever as long as the Stoppable interface is passed so if another thread has stopped or aborted the hosting service, we'll notice and give up trying. Doing the latter is probably better than some kinda timeout. HBASE-3062 adds a timed retry on the first zk operation. This issue is about generalizing what is over there across all zk access. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4147) StoreFile query usage report
[ https://issues.apache.org/jira/browse/HBASE-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072807#comment-13072807 ] Doug Meil commented on HBASE-4147: -- I think that instrumenting StoreFileScanner by gathering time spent for all the 'next' and 'seek' calls would do it. And then on 'close' it would publish the detailed record to some internal service that would gather up all the these detail records and then periodically dump the summary. I'm doing some hand-waving here because we don't want to introduce concurrency issues in the publishing process (e.g., publishing to something that is synchronized will effectively single-thread StoreFileScanners which would be a non-starter), but based on my understanding of the code it seems like this would be a fairly targeted change. Thoughts? StoreFile query usage report Key: HBASE-4147 URL: https://issues.apache.org/jira/browse/HBASE-4147 Project: HBase Issue Type: Improvement Reporter: Doug Meil Priority: Minor Detailed information on what HBase is doing in terms of reads is hard to come by. What would be useful is to have a periodic StoreFile query report. Specifically, this could run on a configured interval (e.g., every 30 seconds, 60 seconds) and dump the output to the log files. This would have all StoreFiles accessed during the reporting period (and with the Path we would also know region, CF, and table), # of times the StoreFile was accessed, the size of the StoreFile, and the total time (ms) spent processing that StoreFile. Even this level of summary would be useful to detect a which tables CFs are being accessed the most, and including the StoreFile would provide insight into relative uncompaction (i.e., lots of StoreFiles). I think the log-output, as opposed to UI, is an important facet with this. I'm assuming that users will slice and dice this data on their own so I think we should skip any kind of admin view for now (i.e., new JSPs, new APIs to expose this data). Just getting this to log-file would be a big improvement. Will this have a non-zero performance impact? Yes. Hopefully small, but yes it will. However, flying a plane without any instrumentation isn't fun. :-) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072809#comment-13072809 ] Doug Meil commented on HBASE-4089: -- Regarding dumping the summary report to the log, I think exposing a public 'printSummary' (logSummary?) method on LruBlockCache would do it. Another thread can take care of the scheduling on how often the block cache summary should be run. blockCache contents report -- Key: HBASE-4089 URL: https://issues.apache.org/jira/browse/HBASE-4089 Project: HBase Issue Type: New Feature Reporter: Doug Meil Summarized block-cache report for a RegionServer would be helpful. For example ... table1 cf1 100 blocks, totalBytes=y, averageTimeInCache= hours cf2 200 blocks, totalBytes=z, averageTimeInCache= hours table2 cf1 75 blocks, totalBytes=y, averageTimeInCache= hours cf2 150 blocks, totalBytes=z, averageTimeInCache= hours ... Etc. The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important. No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective. Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4089) blockCache contents report
[ https://issues.apache.org/jira/browse/HBASE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072812#comment-13072812 ] Doug Meil commented on HBASE-4089: -- If this approach is acceptable, probably should add this to the BlockCache interface. This is how the block cache is accessed. blockCache contents report -- Key: HBASE-4089 URL: https://issues.apache.org/jira/browse/HBASE-4089 Project: HBase Issue Type: New Feature Reporter: Doug Meil Summarized block-cache report for a RegionServer would be helpful. For example ... table1 cf1 100 blocks, totalBytes=y, averageTimeInCache= hours cf2 200 blocks, totalBytes=z, averageTimeInCache= hours table2 cf1 75 blocks, totalBytes=y, averageTimeInCache= hours cf2 150 blocks, totalBytes=z, averageTimeInCache= hours ... Etc. The current metrics list blockCacheSize and blockCacheFree, but there is no way to know what's in there. Any single block isn't really important, but the patterns of what CF/Table they came from, how big are they, and how long (on average) they've been in the cache, are important. No such interface exists in HRegionInterface. But I think it would be helpful from an operational perspective. Updated (7-29): Removing suggestion for UI. I would be happy just to get this report on a configured interval dumped to a log file. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss
[
https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072813#comment-13072813
]
Ted Yu commented on HBASE-3065:
---
With the addendum, TestSplitLogManager passed and I got one fewer test failure
from TestDistributedLogSplitting:
{code}
Failed tests:
testThreeRSAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting)
testWorkerAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting)
{code}
Applied addendum to TRUNK.
Thanks for the analysis Ramkrishna.
Retry all 'retryable' zk operations; e.g. connection loss
-
Key: HBASE-3065
URL: https://issues.apache.org/jira/browse/HBASE-3065
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: Liyin Tang
Priority: Critical
Fix For: 0.92.0
Attachments: 3065-v3.txt, 3065-v4.txt, HBASE-3065-addendum.patch,
HBase-3065[r1088475]_1.patch, hbase3065_2.patch
The 'new' master refactored our zk code tidying up all zk accesses and
coralling them behind nice zk utility classes. One improvement was letting
out all KeeperExceptions letting the client deal. Thats good generally
because in old days, we'd suppress important state zk changes in state. But
there is at least one case the new zk utility could handle for the
application and thats the class of retryable KeeperExceptions. The one that
comes to mind is conection loss. On connection loss we should retry the
just-failed operation. Usually the retry will just work. At worse, on
reconnect, we'll pick up the expired session event.
Adding in this change shouldn't be too bad given the refactor of zk corralled
all zk access into one or two classes only.
One thing to consider though is how much we should retry. We could retry on
a timer or we could retry for ever as long as the Stoppable interface is
passed so if another thread has stopped or aborted the hosting service, we'll
notice and give up trying. Doing the latter is probably better than some
kinda timeout.
HBASE-3062 adds a timed retry on the first zk operation. This issue is about
generalizing what is over there across all zk access.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4144) RS does not abort if the initialization of RS fails
[
https://issues.apache.org/jira/browse/HBASE-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-4144:
--
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
RS does not abort if the initialization of RS fails
Key: HBASE-4144
URL: https://issues.apache.org/jira/browse/HBASE-4144
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.3
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Attachments: HBASE-4144_0.90.patch, HBASE-4144_trunk.patch
If any exception occurs while initialization of RS the RS doesnot get
aborted whereas only the RPC server gets stopped.
private void preRegistrationInitialization()
throws IOException, InterruptedException {
try {
initializeZooKeeper();
initializeThreads();
int nbBlocks = conf.getInt(hbase.regionserver.nbreservationblocks,4);
for (int i = 0; i nbBlocks; i++) {
reservedSpace.add(new
byte[HConstants.DEFAULT_SIZE_RESERVATION_BLOCK]);
}
} catch (Throwable t) {
// Call stop if error or process will stick around for ever since
server
// puts up non-daemon threads.
LOG.error(Stopping HRS because failed initialize, t);
this.rpcServer.stop();
}
}
So if any exception occurs while initilization the RPC server gets stopped
but RS process is still running. But the log says stopping HRegionServer.
So in the below code the catch() block will be executed when the RPCServer
stop fails?
In all other cases it doesnt handle any initialization failure.
try {
// Do pre-registration initializations; zookeeper, lease threads,tc.
preRegistrationInitialization();
} catch (Exception e) {
abort(Fatal exception during initialization, e);
}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4138) If zookeeper.znode.parent is not specifed explicitly in Client code then HTable object loops continuously waiting for the root region by using /hbase as the base node.
[ https://issues.apache.org/jira/browse/HBASE-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072826#comment-13072826 ] Ted Yu commented on HBASE-4138: --- Integrated to TRUNK. Thanks for the patch Ramkrishna. If zookeeper.znode.parent is not specifed explicitly in Client code then HTable object loops continuously waiting for the root region by using /hbase as the base node. --- Key: HBASE-4138 URL: https://issues.apache.org/jira/browse/HBASE-4138 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.90.3 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.92.0 Attachments: HBASE-4138_trunk_1.patch, HBASE-4138_trunk_2.patch, HBASE-4138_trunk_3.patch Change the zookeeper.znode.parent property (default is /hbase). Now do not specify this change in the client code. Use the HTable Object. The HTable is not able to find the root region and keeps continuously looping. Find the stack trace: Object.wait(long) line: not available [native method] RootRegionTracker(ZooKeeperNodeTracker).blockUntilAvailable(long) line: 122 RootRegionTracker.waitRootRegionLocation(long) line: 73 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[], boolean) line: 578 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[]) line: 558 HConnectionManager$HConnectionImplementation.locateRegionInMeta(byte[], byte[], byte[], boolean, Object) line: 687 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[], boolean) line: 589 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[]) line: 558 HConnectionManager$HConnectionImplementation.locateRegionInMeta(byte[], byte[], byte[], boolean, Object) line: 687 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[], boolean) line: 593 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[]) line: 558 HTable.init(Configuration, byte[]) line: 171 HTable.init(Configuration, String) line: 145 HBaseTest.test() line: 45 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss
[ https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072829#comment-13072829 ] ramkrishna.s.vasudevan commented on HBASE-3065: --- @Ted The testcase testWorkerAbort() is working fine. testThreeRSAbort I am getting 'java.lang.AssertionError: expected:4000 but was:3400.' Is this the error that we get? Retry all 'retryable' zk operations; e.g. connection loss - Key: HBASE-3065 URL: https://issues.apache.org/jira/browse/HBASE-3065 Project: HBase Issue Type: Bug Reporter: stack Assignee: Liyin Tang Priority: Critical Fix For: 0.92.0 Attachments: 3065-v3.txt, 3065-v4.txt, HBASE-3065-addendum.patch, HBase-3065[r1088475]_1.patch, hbase3065_2.patch The 'new' master refactored our zk code tidying up all zk accesses and coralling them behind nice zk utility classes. One improvement was letting out all KeeperExceptions letting the client deal. Thats good generally because in old days, we'd suppress important state zk changes in state. But there is at least one case the new zk utility could handle for the application and thats the class of retryable KeeperExceptions. The one that comes to mind is conection loss. On connection loss we should retry the just-failed operation. Usually the retry will just work. At worse, on reconnect, we'll pick up the expired session event. Adding in this change shouldn't be too bad given the refactor of zk corralled all zk access into one or two classes only. One thing to consider though is how much we should retry. We could retry on a timer or we could retry for ever as long as the Stoppable interface is passed so if another thread has stopped or aborted the hosting service, we'll notice and give up trying. Doing the latter is probably better than some kinda timeout. HBASE-3062 adds a timed retry on the first zk operation. This issue is about generalizing what is over there across all zk access. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss
[ https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HBASE-3065: --- There is still more to fix. Retry all 'retryable' zk operations; e.g. connection loss - Key: HBASE-3065 URL: https://issues.apache.org/jira/browse/HBASE-3065 Project: HBase Issue Type: Bug Reporter: stack Assignee: Liyin Tang Priority: Critical Fix For: 0.92.0 Attachments: 3065-v3.txt, 3065-v4.txt, HBASE-3065-addendum.patch, HBase-3065[r1088475]_1.patch, hbase3065_2.patch The 'new' master refactored our zk code tidying up all zk accesses and coralling them behind nice zk utility classes. One improvement was letting out all KeeperExceptions letting the client deal. Thats good generally because in old days, we'd suppress important state zk changes in state. But there is at least one case the new zk utility could handle for the application and thats the class of retryable KeeperExceptions. The one that comes to mind is conection loss. On connection loss we should retry the just-failed operation. Usually the retry will just work. At worse, on reconnect, we'll pick up the expired session event. Adding in this change shouldn't be too bad given the refactor of zk corralled all zk access into one or two classes only. One thing to consider though is how much we should retry. We could retry on a timer or we could retry for ever as long as the Stoppable interface is passed so if another thread has stopped or aborted the hosting service, we'll notice and give up trying. Doing the latter is probably better than some kinda timeout. HBASE-3062 adds a timed retry on the first zk operation. This issue is about generalizing what is over there across all zk access. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4138) If zookeeper.znode.parent is not specifed explicitly in Client code then HTable object loops continuously waiting for the root region by using /hbase as the base node.
[ https://issues.apache.org/jira/browse/HBASE-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072882#comment-13072882 ] Hudson commented on HBASE-4138: --- Integrated in HBase-TRUNK #2063 (See [https://builds.apache.org/job/HBase-TRUNK/2063/]) HBASE-4138 If zookeeper.znode.parent is not specifed explicitly in Client code then HTable object loops continuously waiting for the root region by using /hbase as the base node.(ramkrishna.s.vasudevan) tedyu : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKTable.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java If zookeeper.znode.parent is not specifed explicitly in Client code then HTable object loops continuously waiting for the root region by using /hbase as the base node. --- Key: HBASE-4138 URL: https://issues.apache.org/jira/browse/HBASE-4138 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.90.3 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.92.0 Attachments: HBASE-4138_trunk_1.patch, HBASE-4138_trunk_2.patch, HBASE-4138_trunk_3.patch Change the zookeeper.znode.parent property (default is /hbase). Now do not specify this change in the client code. Use the HTable Object. The HTable is not able to find the root region and keeps continuously looping. Find the stack trace: Object.wait(long) line: not available [native method] RootRegionTracker(ZooKeeperNodeTracker).blockUntilAvailable(long) line: 122 RootRegionTracker.waitRootRegionLocation(long) line: 73 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[], boolean) line: 578 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[]) line: 558 HConnectionManager$HConnectionImplementation.locateRegionInMeta(byte[], byte[], byte[], boolean, Object) line: 687 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[], boolean) line: 589 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[]) line: 558 HConnectionManager$HConnectionImplementation.locateRegionInMeta(byte[], byte[], byte[], boolean, Object) line: 687 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[], boolean) line: 593 HConnectionManager$HConnectionImplementation.locateRegion(byte[], byte[]) line: 558 HTable.init(Configuration, byte[]) line: 171 HTable.init(Configuration, String) line: 145 HBaseTest.test() line: 45 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss
[ https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072881#comment-13072881 ] Hudson commented on HBASE-3065: --- Integrated in HBase-TRUNK #2063 (See [https://builds.apache.org/job/HBase-TRUNK/2063/]) HBASE-3065 Addendum that removes metadata in getDataSetWatchSuccess() (Ramkrishna) tedyu : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java Retry all 'retryable' zk operations; e.g. connection loss - Key: HBASE-3065 URL: https://issues.apache.org/jira/browse/HBASE-3065 Project: HBase Issue Type: Bug Reporter: stack Assignee: Liyin Tang Priority: Critical Fix For: 0.92.0 Attachments: 3065-v3.txt, 3065-v4.txt, HBASE-3065-addendum.patch, HBase-3065[r1088475]_1.patch, hbase3065_2.patch The 'new' master refactored our zk code tidying up all zk accesses and coralling them behind nice zk utility classes. One improvement was letting out all KeeperExceptions letting the client deal. Thats good generally because in old days, we'd suppress important state zk changes in state. But there is at least one case the new zk utility could handle for the application and thats the class of retryable KeeperExceptions. The one that comes to mind is conection loss. On connection loss we should retry the just-failed operation. Usually the retry will just work. At worse, on reconnect, we'll pick up the expired session event. Adding in this change shouldn't be too bad given the refactor of zk corralled all zk access into one or two classes only. One thing to consider though is how much we should retry. We could retry on a timer or we could retry for ever as long as the Stoppable interface is passed so if another thread has stopped or aborted the hosting service, we'll notice and give up trying. Doing the latter is probably better than some kinda timeout. HBASE-3062 adds a timed retry on the first zk operation. This issue is about generalizing what is over there across all zk access. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss
[ https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072886#comment-13072886 ] stack commented on HBASE-3065: -- Excellent Ram. I'd figured the first part of your patch last night -- the metdata prefix -- but not the second. Thank you. Let me build on your patch for the rest of the fix. So, yes, distributed log splitting does heavy interactions with zk. It went in after the original 3065 patch was done. The url encoding that distributed splitting zk'ing does is clashing with suffixes that this patch adds to file names -- we need to find the places where we want to use znode name and make sure we url decode. The second issue is the one you note above where this patch adds metadata at front of data and we need to strip it reading in a few places. Let me see how far I get today (I'm gone for a week starting this evening...) Retry all 'retryable' zk operations; e.g. connection loss - Key: HBASE-3065 URL: https://issues.apache.org/jira/browse/HBASE-3065 Project: HBase Issue Type: Bug Reporter: stack Assignee: Liyin Tang Priority: Critical Fix For: 0.92.0 Attachments: 3065-v3.txt, 3065-v4.txt, HBASE-3065-addendum.patch, HBase-3065[r1088475]_1.patch, hbase3065_2.patch The 'new' master refactored our zk code tidying up all zk accesses and coralling them behind nice zk utility classes. One improvement was letting out all KeeperExceptions letting the client deal. Thats good generally because in old days, we'd suppress important state zk changes in state. But there is at least one case the new zk utility could handle for the application and thats the class of retryable KeeperExceptions. The one that comes to mind is conection loss. On connection loss we should retry the just-failed operation. Usually the retry will just work. At worse, on reconnect, we'll pick up the expired session event. Adding in this change shouldn't be too bad given the refactor of zk corralled all zk access into one or two classes only. One thing to consider though is how much we should retry. We could retry on a timer or we could retry for ever as long as the Stoppable interface is passed so if another thread has stopped or aborted the hosting service, we'll notice and give up trying. Doing the latter is probably better than some kinda timeout. HBASE-3062 adds a timed retry on the first zk operation. This issue is about generalizing what is over there across all zk access. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3581) hbase rpc should send size of response
[ https://issues.apache.org/jira/browse/HBASE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-3581: -- Fix Version/s: (was: 0.92.0) 0.94.0 Moving to 0.94 for now. hbase rpc should send size of response -- Key: HBASE-3581 URL: https://issues.apache.org/jira/browse/HBASE-3581 Project: HBase Issue Type: Improvement Reporter: ryan rawson Assignee: ryan rawson Priority: Critical Fix For: 0.94.0 Attachments: HBASE-rpc-response.txt The RPC reply from Server-Client does not include the size of the payload, it is framed like so: i32 callId byte errorFlag byte[] data The data segment would contain enough info about how big the response is so that it could be decoded by a writable reader. This makes it difficult to write buffering clients, who might read the entire 'data' then pass it to a decoder. While less memory efficient, if you want to easily write block read clients (eg: nio) it would be necessary to send the size along so that the client could snarf into a local buf. The new proposal is: i32 callId i32 size byte errorFlag byte[] data the size being sizeof(data) + sizeof(errorFlag). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss
[ https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3065: - Priority: Blocker (was: Critical) Retry all 'retryable' zk operations; e.g. connection loss - Key: HBASE-3065 URL: https://issues.apache.org/jira/browse/HBASE-3065 Project: HBase Issue Type: Bug Reporter: stack Assignee: Liyin Tang Priority: Blocker Fix For: 0.92.0 Attachments: 3065-v3.txt, 3065-v4.txt, HBASE-3065-addendum.patch, HBase-3065[r1088475]_1.patch, hbase3065_2.patch The 'new' master refactored our zk code tidying up all zk accesses and coralling them behind nice zk utility classes. One improvement was letting out all KeeperExceptions letting the client deal. Thats good generally because in old days, we'd suppress important state zk changes in state. But there is at least one case the new zk utility could handle for the application and thats the class of retryable KeeperExceptions. The one that comes to mind is conection loss. On connection loss we should retry the just-failed operation. Usually the retry will just work. At worse, on reconnect, we'll pick up the expired session event. Adding in this change shouldn't be too bad given the refactor of zk corralled all zk access into one or two classes only. One thing to consider though is how much we should retry. We could retry on a timer or we could retry for ever as long as the Stoppable interface is passed so if another thread has stopped or aborted the hosting service, we'll notice and give up trying. Doing the latter is probably better than some kinda timeout. HBASE-3062 adds a timed retry on the first zk operation. This issue is about generalizing what is over there across all zk access. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4015) Refactor the TimeoutMonitor to make it less racy
[ https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4015: - Priority: Blocker (was: Critical) Refactor the TimeoutMonitor to make it less racy Key: HBASE-4015 URL: https://issues.apache.org/jira/browse/HBASE-4015 Project: HBase Issue Type: Sub-task Affects Versions: 0.90.3 Reporter: Jean-Daniel Cryans Priority: Blocker Fix For: 0.92.0 The current implementation of the TimeoutMonitor acts like a race condition generator, mostly making things worse rather than better. It does it's own thing for a while without caring for what's happening in the rest of the master. The first thing that needs to happen is that the regions should not be processed in one big batch, because that sometimes can take minutes to process (meanwhile a region that timed out opening might have opened, then what happens is it will be reassigned by the TimeoutMonitor generating the never ending PENDING_OPEN situation). Those operations should also be done more atomically, although I'm not sure how to do it in a scalable way in this case. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-1621) merge tool should work on online cluster, but disabled table
[ https://issues.apache.org/jira/browse/HBASE-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-1621: - Priority: Blocker (was: Critical) Need this so can do improvements to hbck --fix (The improvements to hbck --fix) can happen out of band with 0.92 release but need this in place) merge tool should work on online cluster, but disabled table Key: HBASE-1621 URL: https://issues.apache.org/jira/browse/HBASE-1621 Project: HBase Issue Type: Bug Reporter: ryan rawson Assignee: stack Priority: Blocker Fix For: 0.92.0 Attachments: 1621-trunk.txt, HBASE-1621-v2.patch, HBASE-1621.patch, hbase-onlinemerge.patch taking down the entire cluster to merge 2 regions is a pain, i dont see why the table or regions specifically couldnt be taken offline, then merged then brought back up. this might need a new API to the regionservers so they can take direction from not just the master. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss
[ https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072906#comment-13072906 ] Ted Yu commented on HBASE-3065: --- I guess Prakash and Liyin would be able to accommodate this JIRA for distributed log splitting since they work together. Retry all 'retryable' zk operations; e.g. connection loss - Key: HBASE-3065 URL: https://issues.apache.org/jira/browse/HBASE-3065 Project: HBase Issue Type: Bug Reporter: stack Assignee: Liyin Tang Priority: Blocker Fix For: 0.92.0 Attachments: 3065-v3.txt, 3065-v4.txt, HBASE-3065-addendum.patch, HBase-3065[r1088475]_1.patch, hbase3065_2.patch The 'new' master refactored our zk code tidying up all zk accesses and coralling them behind nice zk utility classes. One improvement was letting out all KeeperExceptions letting the client deal. Thats good generally because in old days, we'd suppress important state zk changes in state. But there is at least one case the new zk utility could handle for the application and thats the class of retryable KeeperExceptions. The one that comes to mind is conection loss. On connection loss we should retry the just-failed operation. Usually the retry will just work. At worse, on reconnect, we'll pick up the expired session event. Adding in this change shouldn't be too bad given the refactor of zk corralled all zk access into one or two classes only. One thing to consider though is how much we should retry. We could retry on a timer or we could retry for ever as long as the Stoppable interface is passed so if another thread has stopped or aborted the hosting service, we'll notice and give up trying. Doing the latter is probably better than some kinda timeout. HBASE-3062 adds a timed retry on the first zk operation. This issue is about generalizing what is over there across all zk access. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3581) hbase rpc should send size of response
[ https://issues.apache.org/jira/browse/HBASE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3581: - Fix Version/s: (was: 0.94.0) 0.92.0 hbase rpc should send size of response -- Key: HBASE-3581 URL: https://issues.apache.org/jira/browse/HBASE-3581 Project: HBase Issue Type: Improvement Reporter: ryan rawson Assignee: ryan rawson Priority: Critical Fix For: 0.92.0 Attachments: HBASE-rpc-response.txt The RPC reply from Server-Client does not include the size of the payload, it is framed like so: i32 callId byte errorFlag byte[] data The data segment would contain enough info about how big the response is so that it could be decoded by a writable reader. This makes it difficult to write buffering clients, who might read the entire 'data' then pass it to a decoder. While less memory efficient, if you want to easily write block read clients (eg: nio) it would be necessary to send the size along so that the client could snarf into a local buf. The new proposal is: i32 callId i32 size byte errorFlag byte[] data the size being sizeof(data) + sizeof(errorFlag). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3581) hbase rpc should send size of response
[ https://issues.apache.org/jira/browse/HBASE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072916#comment-13072916 ] stack commented on HBASE-3581: -- ugh we should do this for tsdb and asynchbase I'm going to pull it in again (sorry Ted)... we can punt later if it don't make cut. hbase rpc should send size of response -- Key: HBASE-3581 URL: https://issues.apache.org/jira/browse/HBASE-3581 Project: HBase Issue Type: Improvement Reporter: ryan rawson Assignee: ryan rawson Priority: Critical Fix For: 0.92.0 Attachments: HBASE-rpc-response.txt The RPC reply from Server-Client does not include the size of the payload, it is framed like so: i32 callId byte errorFlag byte[] data The data segment would contain enough info about how big the response is so that it could be decoded by a writable reader. This makes it difficult to write buffering clients, who might read the entire 'data' then pass it to a decoder. While less memory efficient, if you want to easily write block read clients (eg: nio) it would be necessary to send the size along so that the client could snarf into a local buf. The new proposal is: i32 callId i32 size byte errorFlag byte[] data the size being sizeof(data) + sizeof(errorFlag). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072940#comment-13072940 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1211 --- conf/hbase-env.sh https://reviews.apache.org/r/1214/#comment2675 Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly a bit above the off heap cache size is) src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java https://reviews.apache.org/r/1214/#comment2676 2011 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java https://reviews.apache.org/r/1214/#comment2677 whitespace src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java https://reviews.apache.org/r/1214/#comment2678 license src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java https://reviews.apache.org/r/1214/#comment2679 class comment src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment2680 whitespace here and throughout this file src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment2681 Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc). If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks) src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment2682 longer than 80 chars src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment2683 This seems like a behavior that we may not always want. If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?) we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap) src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment2684 this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java https://reviews.apache.org/r/1214/#comment2685 huh? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java https://reviews.apache.org/r/1214/#comment2686 line 80 chars src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java https://reviews.apache.org/r/1214/#comment2687 getTotalNumBlocks() and getRemainingNumBlocks() or something? i find the method names a little unclear (or just add some javadoc) src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java https://reviews.apache.org/r/1214/#comment2688 javadoc on these src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java https://reviews.apache.org/r/1214/#comment2689 I'm not totally clear on why the SlabCache contains a bunch of SingleSizeCaches. Why do you need to layer BlockCaches on top of BlockCaches? You'll have one slab per size rather than one cache per size? Can you not pass the right evictor callback in so it goes back to the right slab? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java https://reviews.apache.org/r/1214/#comment2690 Why these ratios? At the least, this should all be configurable (even if just in code and undocumented). Do we need to always pre-allocate everything and determine the block/slab sizes and all that? The design seems inflexible because it's all determine during construction rather than being adaptive. I'm okay with the first
[jira] [Commented] (HBASE-3581) hbase rpc should send size of response
[ https://issues.apache.org/jira/browse/HBASE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072951#comment-13072951 ] ryan rawson commented on HBASE-3581: Lets just commit the patch as is. Optimizing the deployment complexities of users isn't our job, and at the same time it's preventing a extremely useful patch from going in. Take the flag day, and in the future everything else will be easier. (until there is another flag day, but you want a job dont you?) hbase rpc should send size of response -- Key: HBASE-3581 URL: https://issues.apache.org/jira/browse/HBASE-3581 Project: HBase Issue Type: Improvement Reporter: ryan rawson Assignee: ryan rawson Priority: Critical Fix For: 0.92.0 Attachments: HBASE-rpc-response.txt The RPC reply from Server-Client does not include the size of the payload, it is framed like so: i32 callId byte errorFlag byte[] data The data segment would contain enough info about how big the response is so that it could be decoded by a writable reader. This makes it difficult to write buffering clients, who might read the entire 'data' then pass it to a decoder. While less memory efficient, if you want to easily write block read clients (eg: nio) it would be necessary to send the size along so that the client could snarf into a local buf. The new proposal is: i32 callId i32 size byte errorFlag byte[] data the size being sizeof(data) + sizeof(errorFlag). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3065) Retry all 'retryable' zk operations; e.g. connection loss
[ https://issues.apache.org/jira/browse/HBASE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072952#comment-13072952 ] stack commented on HBASE-3065: -- @Ram We could add methods to RecoverableZookeeper than wrapped the callbacks to strip the metadata prefix; maybe we should do that sometime. The addendum will do for now. Retry all 'retryable' zk operations; e.g. connection loss - Key: HBASE-3065 URL: https://issues.apache.org/jira/browse/HBASE-3065 Project: HBase Issue Type: Bug Reporter: stack Assignee: Liyin Tang Priority: Blocker Fix For: 0.92.0 Attachments: 3065-v3.txt, 3065-v4.txt, HBASE-3065-addendum.patch, HBase-3065[r1088475]_1.patch, hbase3065_2.patch The 'new' master refactored our zk code tidying up all zk accesses and coralling them behind nice zk utility classes. One improvement was letting out all KeeperExceptions letting the client deal. Thats good generally because in old days, we'd suppress important state zk changes in state. But there is at least one case the new zk utility could handle for the application and thats the class of retryable KeeperExceptions. The one that comes to mind is conection loss. On connection loss we should retry the just-failed operation. Usually the retry will just work. At worse, on reconnect, we'll pick up the expired session event. Adding in this change shouldn't be too bad given the refactor of zk corralled all zk access into one or two classes only. One thing to consider though is how much we should retry. We could retry on a timer or we could retry for ever as long as the Stoppable interface is passed so if another thread has stopped or aborted the hosting service, we'll notice and give up trying. Doing the latter is probably better than some kinda timeout. HBASE-3062 adds a timed retry on the first zk operation. This issue is about generalizing what is over there across all zk access. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4032) HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
[
https://issues.apache.org/jira/browse/HBASE-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4032:
-
Attachment: 4032-v2.txt
Fix setTableDesc to do as it used to. Added a unit test to prove that these
new methods work as we'd expect.
HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
---
Key: HBASE-4032
URL: https://issues.apache.org/jira/browse/HBASE-4032
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Assignee: stack
Priority: Blocker
Fix For: 0.92.0
Attachments: 4032-v2.txt, 4032.txt
After HBASE-451, HRegionInfo#getTableDesc has been modified to always return
{{null}}.
One immediate effect is broken unit tests.
That aside, it is not in the spirit of deprecation to actually break the
method until after the deprecation cycle, it's a bug.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4032) HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
[
https://issues.apache.org/jira/browse/HBASE-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072993#comment-13072993
]
stack commented on HBASE-4032:
--
Need a review please (and someone to commit if +1)
HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
---
Key: HBASE-4032
URL: https://issues.apache.org/jira/browse/HBASE-4032
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Assignee: stack
Priority: Blocker
Fix For: 0.92.0
Attachments: 4032-v2.txt, 4032.txt
After HBASE-451, HRegionInfo#getTableDesc has been modified to always return
{{null}}.
One immediate effect is broken unit tests.
That aside, it is not in the spirit of deprecation to actually break the
method until after the deprecation cycle, it's a bug.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3446) ProcessServerShutdown fails if META moves, orphaning lots of regions
[ https://issues.apache.org/jira/browse/HBASE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072994#comment-13072994 ] stack commented on HBASE-3446: -- So, I need to update the last patch here and work on the failures seen. I want to write a test too to prove that we have retries after this patch goes in. ProcessServerShutdown fails if META moves, orphaning lots of regions Key: HBASE-3446 URL: https://issues.apache.org/jira/browse/HBASE-3446 Project: HBase Issue Type: Bug Affects Versions: 0.90.0 Reporter: Todd Lipcon Assignee: stack Priority: Blocker Fix For: 0.92.0 Attachments: 3446-v11.txt, 3446-v12.txt, 3446-v2.txt, 3446-v3.txt, 3446-v4.txt, 3446-v7.txt, 3446-v9.txt, 3446.txt I ran a rolling restart on a 5 node cluster with lots of regions, and afterwards had LOTS of regions left orphaned. The issue appears to be that ProcessServerShutdown failed because the server hosting META was restarted around the same time as another server was being processed -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4148) HFileOutputFormat doesn't fill in TIMERANGE_KEY metadata
HFileOutputFormat doesn't fill in TIMERANGE_KEY metadata Key: HBASE-4148 URL: https://issues.apache.org/jira/browse/HBASE-4148 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.90.3 Reporter: Todd Lipcon Fix For: 0.90.5 When HFiles are flushed through the normal path, they include an attribute TIMERANGE_KEY which can be used to cull HFiles when performing a time-restricted scan. Files produced by HFileOutputFormat are currently missing this metadata. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4148) HFileOutputFormat doesn't fill in TIMERANGE_KEY metadata
[ https://issues.apache.org/jira/browse/HBASE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned HBASE-4148: -- Assignee: Jonathan Hsieh HFileOutputFormat doesn't fill in TIMERANGE_KEY metadata Key: HBASE-4148 URL: https://issues.apache.org/jira/browse/HBASE-4148 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.90.3 Reporter: Todd Lipcon Assignee: Jonathan Hsieh Fix For: 0.90.5 When HFiles are flushed through the normal path, they include an attribute TIMERANGE_KEY which can be used to cull HFiles when performing a time-restricted scan. Files produced by HFileOutputFormat are currently missing this metadata. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4032) HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
[
https://issues.apache.org/jira/browse/HBASE-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073003#comment-13073003
]
Ted Yu commented on HBASE-4032:
---
There is already hbase/regionserver/TestHRegionInfo.java
Shall we name the test as hbase/regionserver/TestTableDescOfHRegionInfo.java ?
HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
---
Key: HBASE-4032
URL: https://issues.apache.org/jira/browse/HBASE-4032
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Assignee: stack
Priority: Blocker
Fix For: 0.92.0
Attachments: 4032-v2.txt, 4032.txt
After HBASE-451, HRegionInfo#getTableDesc has been modified to always return
{{null}}.
One immediate effect is broken unit tests.
That aside, it is not in the spirit of deprecation to actually break the
method until after the deprecation cycle, it's a bug.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4032) HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
[
https://issues.apache.org/jira/browse/HBASE-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073011#comment-13073011
]
stack commented on HBASE-4032:
--
Thats my bad. Copy the method from o.a.h.h.THRI into o.a.h.h.r.THRI?
HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
---
Key: HBASE-4032
URL: https://issues.apache.org/jira/browse/HBASE-4032
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Assignee: stack
Priority: Blocker
Fix For: 0.92.0
Attachments: 4032-v2.txt, 4032.txt
After HBASE-451, HRegionInfo#getTableDesc has been modified to always return
{{null}}.
One immediate effect is broken unit tests.
That aside, it is not in the spirit of deprecation to actually break the
method until after the deprecation cycle, it's a bug.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4032) HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
[
https://issues.apache.org/jira/browse/HBASE-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-4032:
--
Attachment: 4032-v3.txt
Performed minor formatting.
The new test is added to TestHRegionInfo and it passes.
HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
---
Key: HBASE-4032
URL: https://issues.apache.org/jira/browse/HBASE-4032
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Assignee: stack
Priority: Blocker
Fix For: 0.92.0
Attachments: 4032-v2.txt, 4032-v3.txt, 4032.txt
After HBASE-451, HRegionInfo#getTableDesc has been modified to always return
{{null}}.
One immediate effect is broken unit tests.
That aside, it is not in the spirit of deprecation to actually break the
method until after the deprecation cycle, it's a bug.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4032) HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
[
https://issues.apache.org/jira/browse/HBASE-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-4032:
--
Attachment: 4032-v3.txt
HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
---
Key: HBASE-4032
URL: https://issues.apache.org/jira/browse/HBASE-4032
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Assignee: stack
Priority: Blocker
Fix For: 0.92.0
Attachments: 4032-v2.txt, 4032-v3.txt, 4032.txt
After HBASE-451, HRegionInfo#getTableDesc has been modified to always return
{{null}}.
One immediate effect is broken unit tests.
That aside, it is not in the spirit of deprecation to actually break the
method until after the deprecation cycle, it's a bug.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4032) HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
[
https://issues.apache.org/jira/browse/HBASE-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-4032:
--
Attachment: (was: 4032-v3.txt)
HBASE-451 improperly breaks public API HRegionInfo#getTableDesc
---
Key: HBASE-4032
URL: https://issues.apache.org/jira/browse/HBASE-4032
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Assignee: stack
Priority: Blocker
Fix For: 0.92.0
Attachments: 4032-v2.txt, 4032-v3.txt, 4032.txt
After HBASE-451, HRegionInfo#getTableDesc has been modified to always return
{{null}}.
One immediate effect is broken unit tests.
That aside, it is not in the spirit of deprecation to actually break the
method until after the deprecation cycle, it's a bug.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4003) Cleanup Calls Conservatively On Timeout
[
https://issues.apache.org/jira/browse/HBASE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthick Sankarachary updated HBASE-4003:
-
Attachment: HBASE-4003-V2.patch
Cleanup Calls Conservatively On Timeout
---
Key: HBASE-4003
URL: https://issues.apache.org/jira/browse/HBASE-4003
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.90.3
Reporter: Karthick Sankarachary
Assignee: Karthick Sankarachary
Fix For: 0.92.0
Attachments: HBASE-4003-V2.patch, HBASE-4003.patch
In the event of a socket timeout, the {{HBaseClient}} iterates over the
outstanding calls (on that socket), and notifies them that a
{{SocketTimeoutException}} has occurred. Ideally, we should be cleanup up
just those calls that have been outstanding for longer than the specified
socket timeout.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4003) Cleanup Calls Conservatively On Timeout
[
https://issues.apache.org/jira/browse/HBASE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073081#comment-13073081
]
Karthick Sankarachary commented on HBASE-4003:
--
Please review the second version of the patch, which passes most of the tests,
including the one that led to the revert of the initial version. The list of
failed test cases is shown below, which I'm currently looking into:
# org.apache.hadoop.hbase.master.TestDistributedLogSplitting
# org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
# org.apache.hadoop.hbase.TestFullLogReconstruction
# org.apache.hadoop.hbase.client.TestMultiParallel
# org.apache.hadoop.hbase.master.TestMasterFailover
# org.apache.hadoop.hbase.client.TestScannerTimeout
In short, the key changes in the second version are outlined below:
# Ensure that the exception bubbles up in {#cleanupCalls} if RPC timeout is 0,
which will be the case in the even the connection is closed.
# Order the {Connection#calls} map by the call id, so that we can iterate over
it in insertion order.
# Invoke the {#cleanupCalls} method even if the {#receiveResponse} method
doesn't error out. In theory, even if the socket doesn't timeout, it's possible
that some of the older calls may have timed out.
# Reset the socket timeout in {#cleanupCall} to the shortest time that any call
is willing to wait. This is essentially the RPC timeout minus the time the
oldest call has already spent waiting.
Please let me know if you've any questions.
Cleanup Calls Conservatively On Timeout
---
Key: HBASE-4003
URL: https://issues.apache.org/jira/browse/HBASE-4003
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.90.3
Reporter: Karthick Sankarachary
Assignee: Karthick Sankarachary
Fix For: 0.92.0
Attachments: HBASE-4003-V2.patch, HBASE-4003.patch
In the event of a socket timeout, the {{HBaseClient}} iterates over the
outstanding calls (on that socket), and notifies them that a
{{SocketTimeoutException}} has occurred. Ideally, we should be cleanup up
just those calls that have been outstanding for longer than the specified
socket timeout.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3810) Registering a Coprocessor at HTableDescriptor should be less strict
[ https://issues.apache.org/jira/browse/HBASE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073082#comment-13073082 ] jirapos...@reviews.apache.org commented on HBASE-3810: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1051/ --- (Updated 2011-07-30 00:22:53.438496) Review request for hbase. Changes --- Addressed Gary's comments: - Fixed constant naming issues. - Pass conf instance to loadTableCoprocessor() Summary --- Registering a Coprocessor at HTableDescriptor should be less strict - fix regex to account for whitespace around : separators - make path portion optional – we already skip the path handling if the class can be loaded by the classloader - make priority optional and default to USER At revision 3, added HTableDecriptor.addCoprocessor() for loading a table level coprocessor. This addresses bug HBase-3810. https://issues.apache.org/jira/browse/HBase-3810 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 0641f52 src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java c2b3558 src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java e0bde92 Diff: https://reviews.apache.org/r/1051/diff Testing --- Tests passed locally. Thanks, Mingjie Registering a Coprocessor at HTableDescriptor should be less strict --- Key: HBASE-3810 URL: https://issues.apache.org/jira/browse/HBASE-3810 Project: HBase Issue Type: Improvement Components: coprocessors Affects Versions: 0.92.0 Environment: all Reporter: Joerg Schad Assignee: Mingjie Lai Priority: Minor Fix For: 0.92.0 Original Estimate: 2h Remaining Estimate: 2h Registering a Copressor in the following way will fail as the Coprocessor$1 keyword is case sensitive (instead COPROCESSOR$1 works fine). Removing this restriction would improve usability. HTableDescriptor desc = new HTableDescriptor(tName); desc.setValue(Coprocessor$1, path.toString() + : + full_class_name + : + Coprocessor.Priority.USER); -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4003) Cleanup Calls Conservatively On Timeout
[
https://issues.apache.org/jira/browse/HBASE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthick Sankarachary updated HBASE-4003:
-
Status: Patch Available (was: Reopened)
Cleanup Calls Conservatively On Timeout
---
Key: HBASE-4003
URL: https://issues.apache.org/jira/browse/HBASE-4003
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.90.3
Reporter: Karthick Sankarachary
Assignee: Karthick Sankarachary
Fix For: 0.92.0
Attachments: HBASE-4003-V2.patch, HBASE-4003.patch
In the event of a socket timeout, the {{HBaseClient}} iterates over the
outstanding calls (on that socket), and notifies them that a
{{SocketTimeoutException}} has occurred. Ideally, we should be cleanup up
just those calls that have been outstanding for longer than the specified
socket timeout.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073087#comment-13073087 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. conf/hbase-env.sh, line 42 bq. https://reviews.apache.org/r/1214/diff/2/?file=28356#file28356line42 bq. bq. Extra whitespace is obvious on review board. Fixed. bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2 bq. https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line2 bq. bq. Year of copyright. Fixed. bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 48 bq. https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line48 bq. bq. Long line should wrap. Fixed. bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 64 bq. https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line64 bq. bq. Indentation should be corrected for the if block. Fixed bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 2 bq. https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line2 bq. bq. Copyright missing. Fixed bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 25 bq. https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line25 bq. bq. This implies that an eviction may involve fewer than one block. bq. If so, this count can be named evictedBlockCount. Fixed. bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 100 bq. https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100 bq. bq. Should we add hitCount for offHeapCache ? bq. This can be addressed in a follow-on JIRA. I think the offHeapCache should have a hitcount, but we can deal with it later. Right now, it has the full CacheStats metrics. bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115 bq. https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line115 bq. bq. In a follow-on JIRA, we can distinguish between onHeap eviction and offHeap eviction counts. Agreed. I may be redoing this substantially in a follow on Jira. bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 145 bq. https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line145 bq. bq. Is this a typo ? Yup, and fixed. - Li --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1220 --- On 2011-07-28 23:02:50, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-07-28 23:02:50) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa bq.
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073090#comment-13073090
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-07-30 00:39:48.920025)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
Addressed TedYu/JGray's comments.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Pi updated HBASE-4027:
-
Attachment: 4027v7.diff
addressed some of Ted Yu/JGrays comments on reviewboard.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Pi updated HBASE-4027:
-
Attachment: HBase-4027 (1).pdf
New updated benchmark results. Tested with 1gb dataset, as per todd's
suggestion. the FS cache does better, but still can't match the off heap cache.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073097#comment-13073097 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. conf/hbase-env.sh, lines 44-45 bq. https://reviews.apache.org/r/1214/diff/1/?file=28125#file28125line44 bq. bq. Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly a bit above the off heap cache size is) I haven't figured out a way. Just asked StackOverflow - hopefully they'll have an answer. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2 bq. https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line2 bq. bq. 2011 Fixed. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 1 bq. https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line1 bq. bq. license Fixed. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 41 bq. https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line41 bq. bq. whitespace Fixed. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 5 bq. https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line5 bq. bq. class comment Added bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 39 bq. https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line39 bq. bq. Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc). bq. bq. If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks) It can be made more generic easily. I just haven't done it. Will do though - just create a constructor that takes two BlockCaches? bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 66 bq. https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line66 bq. bq. longer than 80 chars fixed. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 77-79 bq. https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line77 bq. bq. This seems like a behavior that we may not always want. bq. bq. If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?) bq. bq. we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap) I was thinking of sending all initial caches to the off heap cache, giving it an inbuilt scan resistance, but LRUBlockCache already does partitioning to deal with scans. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115 bq. https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line115 bq. bq. this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both. The idea was the combined evicted stats of both, but yeah, CacheStats are implemented for both LruBlockCache and SlabCache, so exposing both might be a good idea. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 74 bq. https://reviews.apache.org/r/1214/diff/1/?file=28132#file28132line74 bq. bq. huh? Just a typo. Fixed. Don't know why that line is there. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq.
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073099#comment-13073099
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
bq. On 2011-07-29 05:14:48, Ted Yu wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
line 100
bq. https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100
bq.
bq. Should we add hitCount for offHeapCache ?
bq. This can be addressed in a follow-on JIRA.
bq.
bq. Li Pi wrote:
bq. I think the offHeapCache should have a hitcount, but we can deal with
it later. Right now, it has the full CacheStats metrics.
What I meant to say, is, that it does have a hitcount. It just isn't displayed
anywhere.
- Li
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1220
---
On 2011-07-30 00:39:48, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-07-30 00:39:48)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
ecab7ca
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
150f54f
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
1d5e3fa
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
b600020
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073113#comment-13073113
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1238
---
conf/hbase-env.sh
https://reviews.apache.org/r/1214/#comment2812
Still some whitespaces.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
https://reviews.apache.org/r/1214/#comment2813
Doesn't match actual name of class.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
https://reviews.apache.org/r/1214/#comment2814
I don't see SkipList being used in this class.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
https://reviews.apache.org/r/1214/#comment2815
This doesn't match the actual test.
- Ted
On 2011-07-30 00:39:48, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-07-30 00:39:48)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
ecab7ca
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
150f54f
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
1d5e3fa
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
b600020
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4149) Javadoc for Result.getRow is confusing to new users.
Javadoc for Result.getRow is confusing to new users. Key: HBASE-4149 URL: https://issues.apache.org/jira/browse/HBASE-4149 Project: HBase Issue Type: Bug Reporter: Elliott Clark Priority: Trivial org.apache.hadoop.hbase.client.Result getRow is confusing to new users. The documentation could be read to mean the raw data of the row. In addition it is written with improper grammar. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4149) Javadoc for Result.getRow is confusing to new users.
[ https://issues.apache.org/jira/browse/HBASE-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-4149: - Attachment: HBASE-4149-0.patch Small doc change included. Javadoc for Result.getRow is confusing to new users. Key: HBASE-4149 URL: https://issues.apache.org/jira/browse/HBASE-4149 Project: HBase Issue Type: Bug Reporter: Elliott Clark Priority: Trivial Labels: docuentation Attachments: HBASE-4149-0.patch org.apache.hadoop.hbase.client.Result getRow is confusing to new users. The documentation could be read to mean the raw data of the row. In addition it is written with improper grammar. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4149) Javadoc for Result.getRow is confusing to new users.
[ https://issues.apache.org/jira/browse/HBASE-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-4149: - Status: Patch Available (was: Open) Javadoc for Result.getRow is confusing to new users. Key: HBASE-4149 URL: https://issues.apache.org/jira/browse/HBASE-4149 Project: HBase Issue Type: Bug Reporter: Elliott Clark Priority: Trivial Labels: docuentation Attachments: HBASE-4149-0.patch org.apache.hadoop.hbase.client.Result getRow is confusing to new users. The documentation could be read to mean the raw data of the row. In addition it is written with improper grammar. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3845) data loss because lastSeqWritten can miss memstore edits
[ https://issues.apache.org/jira/browse/HBASE-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Todi updated HBASE-3845: Attachment: HBASE-3845-fix-TestResettingCounters-test.txt Hi folks - I have been working with Prakash The patch I have submitted should fix the issue with TestResettingCounters failing data loss because lastSeqWritten can miss memstore edits Key: HBASE-3845 URL: https://issues.apache.org/jira/browse/HBASE-3845 Project: HBase Issue Type: Bug Affects Versions: 0.90.3 Reporter: Prakash Khemani Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.90.5 Attachments: 0001-HBASE-3845-data-loss-because-lastSeqWritten-can-miss.patch, HBASE-3845-fix-TestResettingCounters-test.txt, HBASE-3845_1.patch, HBASE-3845_2.patch, HBASE-3845_4.patch, HBASE-3845_5.patch, HBASE-3845_6.patch, HBASE-3845__trunk.patch, HBASE-3845_trunk_2.patch, HBASE-3845_trunk_3.patch (I don't have a test case to prove this yet but I have run it by Dhruba and Kannan internally and wanted to put this up for some feedback.) In this discussion let us assume that the region has only one column family. That way I can use region/memstore interchangeably. After a memstore flush it is possible for lastSeqWritten to have a log-sequence-id for a region that is not the earliest log-sequence-id for that region's memstore. HLog.append() does a putIfAbsent into lastSequenceWritten. This is to ensure that we only keep track of the earliest log-sequence-number that is present in the memstore. Every time the memstore is flushed we remove the region's entry in lastSequenceWritten and wait for the next append to populate this entry again. This is where the problem happens. step 1: flusher.prepare() snapshots the memstore under HRegion.updatesLock.writeLock(). step 2 : as soon as the updatesLock.writeLock() is released new entries will be added into the memstore. step 3 : wal.completeCacheFlush() is called. This method removes the region's entry from lastSeqWritten. step 4: the next append will create a new entry for the region in lastSeqWritten(). But this will be the log seq id of the current append. All the edits that were added in step 2 are missing. == as a temporary measure, instead of removing the region's entry in step 3 I will replace it with the log-seq-id of the region-flush-event. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
