Re: Getting unit tests to pass

2013-07-22 Thread Lars Francke
Slightly related, sorry for hijacking: I can't get HBase trunk to build. In particular TestHCM.testClusterStatus always fails for me. I tried on my own Jenkins as well as my IDE (IntelliJ) with the same result (two different machines, CentOS & Mac OS). mvn -U -PrunAllTests -Dmaven.test.redirectTes

[jira] [Created] (HBASE-9025) Thread safety issue in MetaScanner

2013-07-22 Thread Lars Hofhansl (JIRA)
Lars Hofhansl created HBASE-9025: Summary: Thread safety issue in MetaScanner Key: HBASE-9025 URL: https://issues.apache.org/jira/browse/HBASE-9025 Project: HBase Issue Type: Bug Affects

[jira] [Created] (HBASE-9024) TestLogRolling.testLogRollOnPipelineRestart fails/goes zombie

2013-07-22 Thread stack (JIRA)
stack created HBASE-9024: Summary: TestLogRolling.testLogRollOnPipelineRestart fails/goes zombie Key: HBASE-9024 URL: https://issues.apache.org/jira/browse/HBASE-9024 Project: HBase Issue Type: Bug

Re: Getting unit tests to pass

2013-07-22 Thread Stack
nvm. I read the resourcechecker code. It is just printing out before and afters so my speculation that we are up against fd limits is just off. Back to figuring out why tests fail at random St.Ack On Mon, Jul 22, 2013 at 9:50 PM, Stack wrote: > Here is another from tail of > https://iss

Re: Getting unit tests to pass

2013-07-22 Thread Stack
Here is another from tail of https://issues.apache.org/jira/browse/HBASE-5995 2013-07-23 01:23:29,574 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: regionserver.wal.TestLogRolling#testLogRollOnPipelineRestart Thread=39 (was 31) - Thread LEAK? -, OpenFileDescriptor=312 (was 272) - Open

Re: Getting unit tests to pass

2013-07-22 Thread Stack
By way of illustration of how loaded Apache build boxes can be: Thread LEAK? -, OpenFileDescriptor=174 (was 162) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=4 (was 4), SystemLoadAverage=351 (was 383), ProcessCount=142 (was 144), AvailableMemoryMB=819 (was 892), ConnectionCount=0 (was 0

[jira] [Created] (HBASE-9023) TestIOFencing.testFencingAroundCompactionAfterWALSync

2013-07-22 Thread stack (JIRA)
stack created HBASE-9023: Summary: TestIOFencing.testFencingAroundCompactionAfterWALSync Key: HBASE-9023 URL: https://issues.apache.org/jira/browse/HBASE-9023 Project: HBase Issue Type: Bug

RE: does append delete all cell versions except the last one?

2013-07-22 Thread Vladimir Rodionov
Thanks, this is what I thought. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Ted Yu [yuzhih...@gmail.com] Sent: Monday, July 22, 2013 8:07 PM To: dev@hbase.apache.org Subjec

Re: does append delete all cell versions except the last one?

2013-07-22 Thread Ted Yu
In 0.94, from HRegion#append(): // Actually write to Memstore now for (Map.Entry> entry : tempMemstore.entrySet()) { Store store = entry.getKey(); size += store.upsert(entry.getValue()); allKVs.addAll(entry.getValue()); } Looking at MemStore.u

[jira] [Created] (HBASE-9022) TestHLogSplit.testIOEOnOutputThread fails

2013-07-22 Thread stack (JIRA)
stack created HBASE-9022: Summary: TestHLogSplit.testIOEOnOutputThread fails Key: HBASE-9022 URL: https://issues.apache.org/jira/browse/HBASE-9022 Project: HBase Issue Type: Bug Reporter:

RE: does append delete all cell versions except the last one?

2013-07-22 Thread Vladimir Rodionov
Ted, It is not still clear to me. What I am observing in my tests can be explained only if on each Append we read the latest version of a cell, then put delete marker then put new version. The question still remains: expected behavior or a bug? Best regards, Vladimir Rodionov Principal Platform

RE: does append delete all cell versions except the last one?

2013-07-22 Thread Vladimir Rodionov
Here it is: //--- import java.io.IOException; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; import java.util.List; import junit.framework.TestCase; import org.apache.commons.logging.Log; import org.apache.common

[jira] [Created] (HBASE-9021) Create a unit test for ReplicationHLogReaderManager

2013-07-22 Thread Jean-Daniel Cryans (JIRA)
Jean-Daniel Cryans created HBASE-9021: - Summary: Create a unit test for ReplicationHLogReaderManager Key: HBASE-9021 URL: https://issues.apache.org/jira/browse/HBASE-9021 Project: HBase I

[jira] [Reopened] (HBASE-9019) Port HBASE-8690: Reduce unnecessary getFileStatus hdfs calls in TTL hfile and hlog cleanners to 0.94

2013-07-22 Thread Lars Hofhansl (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-9019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reopened HBASE-9019: -- Ted fixed the description. Reopening for the backport. > Port HBASE-8690: Reduce u

Re: Smarter Region assignment policy?

2013-07-22 Thread lars hofhansl
Yep. Seems stale. You can try to comment on it to revive. - Original Message - From: Vladimir Rodionov To: "dev@hbase.apache.org" ; lars hofhansl Cc: Sent: Monday, July 22, 2013 4:21 PM Subject: RE: Smarter Region assignment policy? Is not this https://issues.apache.org/jira/browse/

Re: does append delete all cell versions except the last one?

2013-07-22 Thread Ted Yu
Take a look at the release notes for: https://issues.apache.org/jira/browse/HBASE-4583 The fix went to 0.95.0 Cheers On Mon, Jul 22, 2013 at 4:20 PM, Vladimir Rodionov wrote: > Forgot to mention: 0.94.6.1 (CDH 4.3) > > Best regards, > Vladimir Rodionov > Principal Platform Engineer > Carrier IQ

[question about replication] how to apply delta from Master to Slave after crash ?

2013-07-22 Thread Demai Ni
hi, folks, I am trying to address a recovery scenario after Master crash. In a simple Master - Slave replication setup for t1, and suddenly Master crashes. there will be a small delta of t1 already changed on Master, but haven't been replicated to Slave yet. Assuming Master's filesystem and netw

Re: does append delete all cell versions except the last one?

2013-07-22 Thread Ted Yu
Can you formulate the code below as a unit test so that other people can try it out ? I pasted the code into a unit test but there were a few unresolved method calls such as createGet(). Cheers On Mon, Jul 22, 2013 at 4:20 PM, Vladimir Rodionov wrote: > Forgot to mention: 0.94.6.1 (CDH 4.3) > >

RE: Smarter Region assignment policy?

2013-07-22 Thread Vladimir Rodionov
Is not this https://issues.apache.org/jira/browse/HBASE-6143? Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: lars hofhansl [la...@apache.org] Sent: Monday, July 22, 2013 4:04

RE: does append delete all cell versions except the last one?

2013-07-22 Thread Vladimir Rodionov
Forgot to mention: 0.94.6.1 (CDH 4.3) Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Vladimir Rodionov Sent: Monday, July 22, 2013 4:17 PM To: dev@hbase.apache.org; lars hofha

does append delete all cell versions except the last one?

2013-07-22 Thread Vladimir Rodionov
I am doing some tests of append operation (HTable.append(Append append)) and it seems that if cell being appended has multiple versions , all except the last one get deleted. Is it correct behavior or I am doing something wrong in my tests? The test is simple. I have a row with 3 CF and 3 qualif

Re: Smarter Region assignment policy?

2013-07-22 Thread lars hofhansl
Hmm... So that's an interesting bug then. Might filing a jira? -- Lars - Original Message - From: Vladimir Rodionov To: "dev@hbase.apache.org" ; lars hofhansl Cc: Sent: Monday, July 22, 2013 3:30 PM Subject: RE: Smarter Region assignment policy? OK, you right and I was not. If I do

[jira] [Created] (HBASE-9020) Make hbase-it HBASE_HOME configurable

2013-07-22 Thread Jimmy Xiang (JIRA)
Jimmy Xiang created HBASE-9020: -- Summary: Make hbase-it HBASE_HOME configurable Key: HBASE-9020 URL: https://issues.apache.org/jira/browse/HBASE-9020 Project: HBase Issue Type: Test

RE: Smarter Region assignment policy?

2013-07-22 Thread Vladimir Rodionov
OK, you right and I was not. If I do not disable/enables tables after cluster restart all regions seem get assigned correctly. I think its disabling/enabling tables shuffles regions in a bad way. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vr

[jira] [Resolved] (HBASE-9019) Port HBASE-8690: Reduce unnecessary getFileStatus hdfs calls in TTL hfile and hlog cleanners to 0.94

2013-07-22 Thread stack (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-9019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-9019. -- Resolution: Invalid Resolving issue ill-specified. When I click on the link it takes me to the end of the iss

[jira] [Created] (HBASE-9019) Port HBASE-8690: Reduce unnecessary getFileStatus hdfs calls in TTL hfile and hlog cleanners to 0.94

2013-07-22 Thread Ted Yu (JIRA)
Ted Yu created HBASE-9019: - Summary: Port HBASE-8690: Reduce unnecessary getFileStatus hdfs calls in TTL hfile and hlog cleanners to 0.94 Key: HBASE-9019 URL: https://issues.apache.org/jira/browse/HBASE-9019

[jira] [Resolved] (HBASE-7841) Parallelize offline snapshot in DisabledTableSnapshotHandler

2013-07-22 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-7841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-7841. --- Resolution: Won't Fix > Parallelize offline snapshot in DisabledTableSnapshotHandler > --

[jira] [Resolved] (HBASE-8059) Enhance test-patch.sh so that patch can specify hadoop-2.0 as the default profile

2013-07-22 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-8059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-8059. --- Resolution: Fixed > Enhance test-patch.sh so that patch can specify hadoop-2.0 as the default > profile

[jira] [Resolved] (HBASE-9018) Add timeouts on all tests in TestHLogSplit

2013-07-22 Thread stack (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-9018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-9018. -- Resolution: Fixed Committed to trunk and to 0.95. > Add timeouts on all tests in TestHLogSplit

[jira] [Created] (HBASE-9018) Add timeouts on all tests in TestHLogSplit

2013-07-22 Thread stack (JIRA)
stack created HBASE-9018: Summary: Add timeouts on all tests in TestHLogSplit Key: HBASE-9018 URL: https://issues.apache.org/jira/browse/HBASE-9018 Project: HBase Issue Type: Bug Components

[jira] [Created] (HBASE-9017) Consolidate ChaosMonkey's random objects

2013-07-22 Thread Jimmy Xiang (JIRA)
Jimmy Xiang created HBASE-9017: -- Summary: Consolidate ChaosMonkey's random objects Key: HBASE-9017 URL: https://issues.apache.org/jira/browse/HBASE-9017 Project: HBase Issue Type: Test

[jira] [Created] (HBASE-9016) Cleanup of HRegion (javadoc, unnecessary inits, unnecessary unboxing)

2013-07-22 Thread Alex Newman (JIRA)
Alex Newman created HBASE-9016: -- Summary: Cleanup of HRegion (javadoc, unnecessary inits, unnecessary unboxing) Key: HBASE-9016 URL: https://issues.apache.org/jira/browse/HBASE-9016 Project: HBase

Re: Smarter Region assignment policy?

2013-07-22 Thread lars hofhansl
This is the one: https://issues.apache.org/jira/browse/HBASE-4402 I assume you let the master wait sufficiently to give all RegionServer a chance to sign in? -- Lars From: lars hofhansl To: "dev@hbase.apache.org" Sent: Monday, July 22, 2013 11:55 AM Subject:

Re: Smarter Region assignment policy?

2013-07-22 Thread Elliott Clark
I filed https://issues.apache.org/jira/browse/HBASE-6143 a while ago. Basically there are three methods on the balancer. One that does a real and true balance. There are two others that are used for quick assignment. Those two don't take into account all of the balancing rules and locality info

Getting unit tests to pass

2013-07-22 Thread Stack
Below is a state of hbase 0.95/trunk unit tests (Includes a little taxonomy of test failure type definitions). On Andrew's ec2 build box, 0.95 is passing most of the time: http://54.241.6.143/job/HBase-0.95/ http://54.241.6.143/job/HBase-0.95-Hadoop-2/ It is not as good on Apache build box but i

Re: Smarter Region assignment policy?

2013-07-22 Thread lars hofhansl
I faintly remember there was a jira that attempted to assign the regions to the same region servers after a restart based on existing .META. information (if possible). Will try to find that and see why it is not working as expected. -- Lars From: Vladimir Ro

RE: Smarter Region assignment policy?

2013-07-22 Thread Vladimir Rodionov
Yes, its related but not what I am looking for. Locality index is approaching to 100% after major compaction of all tables. Its often > 90% during regular operation of a cluster, but is far below 50% when cluster restarts. Can assignment manager take into account previous region assignments on n

Re: Smarter Region assignment policy?

2013-07-22 Thread Ted Yu
Please take a look at https://issues.apache.org/jira/browse/HBASE-4755 On Mon, Jul 22, 2013 at 10:56 AM, Vladimir Rodionov wrote: > It seems that current (0.94.6) region assignment on start up policy is not > that smart and does not utilize hdfs block locality > Are there any open HBase JIRA tic

[jira] [Created] (HBASE-9015) Client side coprocessor framework

2013-07-22 Thread Andrew Purtell (JIRA)
Andrew Purtell created HBASE-9015: - Summary: Client side coprocessor framework Key: HBASE-9015 URL: https://issues.apache.org/jira/browse/HBASE-9015 Project: HBase Issue Type: New Feature

Smarter Region assignment policy?

2013-07-22 Thread Vladimir Rodionov
It seems that current (0.94.6) region assignment on start up policy is not that smart and does not utilize hdfs block locality Are there any open HBase JIRA ticket(s) that addresses the issue? Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio

[jira] [Created] (HBASE-9013) Backport HBASE-8994 (Adding log to chaos monkey actions to show what're performed) to 0.94

2013-07-22 Thread Andrew Purtell (JIRA)
Andrew Purtell created HBASE-9013: - Summary: Backport HBASE-8994 (Adding log to chaos monkey actions to show what're performed) to 0.94 Key: HBASE-9013 URL: https://issues.apache.org/jira/browse/HBASE-9013

[jira] [Created] (HBASE-9014) TestHLog.testAppendClose fails

2013-07-22 Thread stack (JIRA)
stack created HBASE-9014: Summary: TestHLog.testAppendClose fails Key: HBASE-9014 URL: https://issues.apache.org/jira/browse/HBASE-9014 Project: HBase Issue Type: Bug Reporter: stack ht

Re: DISCUSS : HFile V3 proposal for tags in 0.96

2013-07-22 Thread Andrew Purtell
[ Reposting from a different account - looks like infra borked LDAP somehow. ] On Fri, Jul 19, 2013 at 4:31 PM, Stack wrote: > 0.96 is way too late already. I am shooting for end of July as cut-off > point when I intend rolling a 0.95.2 RC. > We are shooting for a HFileV3 in time for inclusion

[jira] [Created] (HBASE-9012) TestBlockReorder.testBlockLocationReorder fails

2013-07-22 Thread stack (JIRA)
stack created HBASE-9012: Summary: TestBlockReorder.testBlockLocationReorder fails Key: HBASE-9012 URL: https://issues.apache.org/jira/browse/HBASE-9012 Project: HBase Issue Type: Bug Compo