date:20111024

[
https://issues.apache.org/jira/browse/HBASE-4652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Hsieh updated HBASE-4652:
--

Description:
The suggestions from HBASE-4552 and initial implementations could violate row
atomicity if there is a failure while bulk-loading multiple CF rows. A
mechanism to atomically rollback or commit only upon completion (mvcc at the
store level) is needed if a partial failures encountered when attempting to
bulk load multiple families.

For example, let's say I want to bulk load a region with cfs A, B, C. I issue a
call to an RS region to atomically bulkload the HFiles. The RS loads A and B
successfully but fails on C (hdfs failure, or rs goes down, etc). We could
rollback A and B - if we don't we would have A and B loaded but not C and have
an atomicity violation.

was:
The suggestions from HBASE-4552 and initial implementations could violate row
atomicity if there are multiple CFs. A mechanism to atomically rollback or
commit only upon completion (mvcc at the store level) is needed if a partial
failures encountered when attempting to bulk load multiple families.

Mechanism for atomic recovery from partial failure when atomically
bulk-loading multi-CF rows.
--

Key: HBASE-4652
URL: https://issues.apache.org/jira/browse/HBASE-4652
Project: HBase
Issue Type: Bug
Reporter: Jonathan Hsieh

The suggestions from HBASE-4552 and initial implementations could violate row
atomicity if there is a failure while bulk-loading multiple CF rows. A
mechanism to atomically rollback or commit only upon completion (mvcc at the
store level) is needed if a partial failures encountered when attempting to
bulk load multiple families.
For example, let's say I want to bulk load a region with cfs A, B, C. I issue
a call to an RS region to atomically bulkload the HFiles. The RS loads A and
B successfully but fails on C (hdfs failure, or rs goes down, etc). We could
rollback A and B - if we don't we would have A and B loaded but not C and
have an atomicity violation.

[jira] [Created] (HBASE-4652) Mechanism for atomic recovery from partial failure when atomically bulk-loading multi-CF rows.

2011-10-24 Thread Jonathan Hsieh (Created) (JIRA)

Mechanism for atomic recovery from partial failure when atomically bulk-loading 
multi-CF rows.
--

 Key: HBASE-4652
 URL: https://issues.apache.org/jira/browse/HBASE-4652
 Project: HBase
  Issue Type: Bug
Reporter: Jonathan Hsieh


The suggestions from HBASE-4552 and initial implementations could violate row 
atomicity if there are multiple CFs.  A mechanism to atomically rollback or 
commit only upon completion (mvcc at the store level) is needed if a partial 
failures encountered when attempting to bulk load multiple families.

For example, let's say I want to bulk load a region with cfs A, B, C. I issue a 
call to an RS region to atomically bulkload the HFiles. The RS loads A and B 
successfully but fails on C (hdfs failure, or rs goes down, etc). We could 
rollback A and B - if we don't we would have A and B loaded but not C and have 
an atomicity violation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4552) multi-CF bulk load is not atomic across column families

2011-10-24 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134138#comment-13134138
 ] 

Jonathan Hsieh commented on HBASE-4552:
---

Created recovery mechanism jira at HBASE-4652

 multi-CF bulk load is not atomic across column families
 ---

 Key: HBASE-4552
 URL: https://issues.apache.org/jira/browse/HBASE-4552
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Jonathan Hsieh
 Fix For: 0.92.0


 Currently the bulk load API simply imports one HFile at a time. With 
 multi-column-family support, this is inappropriate, since different CFs show 
 up separately. Instead, the IPC endpoint should take a of CF - HFiles, so we 
 can online them all under a single region-wide lock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4550) When master passed regionserver different address , because regionserver didn't create new zookeeper znode, as a result stop-hbase.sh is hang

2011-10-24 Thread wanbin (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134137#comment-13134137
 ] 

wanbin commented on HBASE-4550:
---

ok.

 When master passed regionserver different address , because regionserver 
 didn't create new zookeeper znode,  as  a result stop-hbase.sh is hang
 ---

 Key: HBASE-4550
 URL: https://issues.apache.org/jira/browse/HBASE-4550
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.90.3
Reporter: wanbin
Assignee: wanbin
 Fix For: 0.90.5

 Attachments: hbase-0.90.3.patch, patch, patch.txt

   Original Estimate: 2h
  Remaining Estimate: 2h

 when master passed regionserver different address, regionserver didn't create 
 new zookeeper znode, master store new address in ServerManager, when call 
 stop-hbase.sh , RegionServerTracker.nodeDeleted received path is old address, 
 serverManager.expireServer is not be called. so stop-hbase.sh is hang.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4649) Add atomic bulk load function to region server

2011-10-24 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134153#comment-13134153
 ] 

Jonathan Hsieh commented on HBASE-4649:
---

review here: https://reviews.apache.org/r/2545/

 Add atomic bulk load function to region server
 --

 Key: HBASE-4649
 URL: https://issues.apache.org/jira/browse/HBASE-4649
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.92.0

 Attachments: 
 0001-HBASE-4649-Add-atomic-bulk-load-function-to-region-s.patch


 Add a method that atomically bulk load multiple hfiles.  Row atomicity 
 guarantees for multi-column family rows require this functionality.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4649) Add atomic bulk load function to region server


 [ 
https://issues.apache.org/jira/browse/HBASE-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4649:
--

Attachment: 0001-HBASE-4649-Add-atomic-bulk-load-function-to-region-s.patch

 Add atomic bulk load function to region server
 --

 Key: HBASE-4649
 URL: https://issues.apache.org/jira/browse/HBASE-4649
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.92.0

 Attachments: 
 0001-HBASE-4649-Add-atomic-bulk-load-function-to-region-s.patch


 Add a method that atomically bulk load multiple hfiles.  Row atomicity 
 guarantees for multi-column family rows require this functionality.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4649) Add atomic bulk load function to region server


 [ 
https://issues.apache.org/jira/browse/HBASE-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4649:
--

Affects Version/s: 0.92.0
   0.90.4
   Status: Patch Available  (was: Open)

 Add atomic bulk load function to region server
 --

 Key: HBASE-4649
 URL: https://issues.apache.org/jira/browse/HBASE-4649
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Affects Versions: 0.90.4, 0.92.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.92.0

 Attachments: 
 0001-HBASE-4649-Add-atomic-bulk-load-function-to-region-s.patch


 Add a method that atomically bulk load multiple hfiles.  Row atomicity 
 guarantees for multi-column family rows require this functionality.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4649) Add atomic bulk load function to region server


 [ 
https://issues.apache.org/jira/browse/HBASE-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4649:
--

Fix Version/s: (was: 0.92.0)

 Add atomic bulk load function to region server
 --

 Key: HBASE-4649
 URL: https://issues.apache.org/jira/browse/HBASE-4649
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Affects Versions: 0.90.4, 0.92.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 
 0001-HBASE-4649-Add-atomic-bulk-load-function-to-region-s.patch


 Add a method that atomically bulk load multiple hfiles.  Row atomicity 
 guarantees for multi-column family rows require this functionality.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4634) test.build.data property overused leading to write data at the wrong place


[ 
https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134171#comment-13134171
 ] 

stack commented on HBASE-4634:
--

@nkeywal  Mind refreshing your TRUNK?  Patch did not apply cleanly and I got 
this when trying to build it:

{code}
[INFO] 
[ERROR] BUILD FAILURE
[INFO] 
[INFO] Compilation failure

/Users/stack/checkout/clean-trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:[265,8]
 cannot find symbol
symbol: variable ShowTime
ShowTime.start();

/Users/stack/checkout/clean-trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:[270,8]
 cannot find symbol
symbol: variable ShowTime
ShowTime.pub(public void run() );

/Users/stack/checkout/clean-trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:[401,4]
 cannot find symbol
symbol  : variable ShowTime
location: class org.apache.hadoop.hbase.master.TestDistributedLogSplitting

/Users/stack/checkout/clean-trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:[403,4]
 cannot find symbol
symbol  : variable ShowTime
location: class org.apache.hadoop.hbase.master.TestDistributedLogSplitting

/Users/stack/checkout/clean-trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:[228,39]
 cannot find symbol
symbol  : method getTestDir()
location: class org.apache.hadoop.hbase.HBaseTestingUtility
{code}

Thanks boss.

 test.build.data property overused leading to write data at the wrong place
 

 Key: HBASE-4634
 URL: https://issues.apache.org/jira/browse/HBASE-4634
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.92.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
 Attachments: 2011020_4634_all.patch, 
 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch, 
 20111022_4634_all.v3.patch, 20111023_4634_all.v4.patch


 test.build.data is overloaded in HBase.At the beginning, it's the Default 
 parent directory for test output., but then it's rewritten to be the 
 directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
 It seems that this value is already used by MiniDFS (i.e. outside of HBase): 
 Name is as it is because mini dfs has hard-codings to put test data here.
 As it is today, there is at least a bug in HBaseTestingUtility:
 {noformat}
   public void initTestDir() {
 if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
   clusterTestBuildDir = setupClusterTestBuildDir();
   System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
 }
   }
 {noformat}
 if you set a value for test.build.data, the test dir will be the parent 
 directory and not a temp subdir, leading to issues as multiple tests will 
 end-ups in the same (bad) directory. This function is barely used today, 
 hence it's not visible, but I would like to use it in some new code.
 A possible fix is to remove the check for null and continue with the 
 overloading, but I don't think it would be a big issue to create a new 
 key(like test.build.data.rootdirectory) specific to the root directory and 
 to use test.build.data only to communicate with MiniDFS. Feedback welcome.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4634) test.build.data property overused leading to write data at the wrong place

2011-10-24 Thread nkeywal (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134181#comment-13134181
 ] 

nkeywal commented on HBASE-4634:


that's an error I missed in my patch. Ok to refresh.

On Mon, Oct 24, 2011 at 5:51 PM, stack (Commented) (JIRA)



 test.build.data property overused leading to write data at the wrong place
 

 Key: HBASE-4634
 URL: https://issues.apache.org/jira/browse/HBASE-4634
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.92.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
 Attachments: 2011020_4634_all.patch, 
 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch, 
 20111022_4634_all.v3.patch, 20111023_4634_all.v4.patch


 test.build.data is overloaded in HBase.At the beginning, it's the Default 
 parent directory for test output., but then it's rewritten to be the 
 directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
 It seems that this value is already used by MiniDFS (i.e. outside of HBase): 
 Name is as it is because mini dfs has hard-codings to put test data here.
 As it is today, there is at least a bug in HBaseTestingUtility:
 {noformat}
   public void initTestDir() {
 if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
   clusterTestBuildDir = setupClusterTestBuildDir();
   System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
 }
   }
 {noformat}
 if you set a value for test.build.data, the test dir will be the parent 
 directory and not a temp subdir, leading to issues as multiple tests will 
 end-ups in the same (bad) directory. This function is barely used today, 
 hence it's not visible, but I would like to use it in some new code.
 A possible fix is to remove the check for null and continue with the 
 overloading, but I don't think it would be a big issue to create a new 
 key(like test.build.data.rootdirectory) specific to the root directory and 
 to use test.build.data only to communicate with MiniDFS. Feedback welcome.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4648) Bytes.toBigDecimal() doesn't use offset


[ 
https://issues.apache.org/jira/browse/HBASE-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134179#comment-13134179
 ] 

stack commented on HBASE-4648:
--

You have a patch please Bryan?

 Bytes.toBigDecimal() doesn't use offset
 ---

 Key: HBASE-4648
 URL: https://issues.apache.org/jira/browse/HBASE-4648
 Project: HBase
  Issue Type: Bug
  Components: util
Affects Versions: 0.90.4
 Environment: Java 1.6.0_26, Mac OS X 10.7 and CentOS 6
Reporter: Bryan Keller

 The Bytes.toBigDecimal(byte[], offset, len) method does not use the offset, 
 thus you will get an incorrect result for the BigDecimal unless the 
 BigDecimal's bytes are at the beginning of the byte array.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4578) NPE when altering a table that has moving regions


[ 
https://issues.apache.org/jira/browse/HBASE-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134227#comment-13134227
 ] 

stack commented on HBASE-4578:
--

bq. // It said that the cluster has 10 regions. but the cluster has 8 regions 
and 2 offlined regions. 

I think the fact that it says 10 regions updated seems fine.  When you say 
offlined, what do you mean?  They are offline parents of splits?  If so, they 
should probably not be counted but no harm done I'd say.  You thinking that we 
should not be altering split, offlined regions?  That sounds good but in 
another issue?

 NPE when altering a table that has moving regions
 -

 Key: HBASE-4578
 URL: https://issues.apache.org/jira/browse/HBASE-4578
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: gaojinchao
Priority: Blocker
 Fix For: 0.92.0

 Attachments: HBASE-4578_Trunk_V1.patch, HBASE-4578_trial_Trunk.patch


 I'm still not a 100% sure on the source of this error, but here's what I was 
 able to get twice while altering a table that was doing a bunch of splits:
 {quote}
 2011-10-11 23:48:59,344 INFO 
 org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
 report); 
 parent=TestTable,0002608338,1318376880454.a75d6815fdfc513fb1c8aabe086c6763. 
 daughter 
 a=TestTable,0002608338,1318376938764.ef170ff6cd8695dc8aec92e542dc9ac1.daughter
  b=TestTable,0003301408,1318376938764.36eb2530341bd46888ede312c5559b5d.
 2011-10-11 23:49:09,579 DEBUG 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Ignoring table not 
 disabled exception for supporting online schema changes.
 2011-10-11 23:49:09,580 INFO 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Handling table 
 operation C_M_MODIFY_TABLE on table TestTable
 2011-10-11 23:49:09,612 INFO org.apache.hadoop.hbase.util.FSUtils: 
 TableInfoPath = hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo tmpPath = 
 hdfs://sv4r11s38:9100/hbase/TestTable/.tmp/.tableinfo.1318376949612
 2011-10-11 23:49:09,692 INFO org.apache.hadoop.hbase.util.FSUtils: 
 TableDescriptor stored. TableInfoPath = 
 hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo
 2011-10-11 23:49:09,693 INFO org.apache.hadoop.hbase.util.FSUtils: Updated 
 tableinfo=hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo to blah
 2011-10-11 23:49:09,695 INFO 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Bucketing regions 
 by region server...
 2011-10-11 23:49:09,695 DEBUG org.apache.hadoop.hbase.client.MetaScanner: 
 Scanning .META. starting at row=TestTable,,00 for max=2147483647 
 rows
 2011-10-11 23:49:09,709 DEBUG 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
 The connection to hconnection-0x132f043bbde02e9 has been closed.
 2011-10-11 23:49:09,709 ERROR org.apache.hadoop.hbase.executor.EventHandler: 
 Caught throwable while processing event C_M_MODIFY_TABLE
 java.lang.NullPointerException
   at java.util.TreeMap.getEntry(TreeMap.java:324)
   at java.util.TreeMap.containsKey(TreeMap.java:209)
   at 
 org.apache.hadoop.hbase.master.handler.TableEventHandler.reOpenAllRegions(TableEventHandler.java:114)
   at 
 org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:90)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:168)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 {quote}
 The first time the shell reported that all the regions were updated 
 correctly, the second time it got stuck for a while:
 {quote}
 6/14 regions updated.
 0/14 regions updated.
 ...
 0/14 regions updated.
 2/16 regions updated.
 ...
 2/16 regions updated.
 8/9 regions updated.
 ...
 8/9 regions updated.
 {quote}
 After which I killed it, redid the alter and it worked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4578) NPE when altering a table that has moving regions


[ 
https://issues.apache.org/jira/browse/HBASE-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134229#comment-13134229
 ] 

stack commented on HBASE-4578:
--

+1 on patch.  Will apply in a little while unless objection.

 NPE when altering a table that has moving regions
 -

 Key: HBASE-4578
 URL: https://issues.apache.org/jira/browse/HBASE-4578
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: gaojinchao
Priority: Blocker
 Fix For: 0.92.0

 Attachments: HBASE-4578_Trunk_V1.patch, HBASE-4578_trial_Trunk.patch


 I'm still not a 100% sure on the source of this error, but here's what I was 
 able to get twice while altering a table that was doing a bunch of splits:
 {quote}
 2011-10-11 23:48:59,344 INFO 
 org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
 report); 
 parent=TestTable,0002608338,1318376880454.a75d6815fdfc513fb1c8aabe086c6763. 
 daughter 
 a=TestTable,0002608338,1318376938764.ef170ff6cd8695dc8aec92e542dc9ac1.daughter
  b=TestTable,0003301408,1318376938764.36eb2530341bd46888ede312c5559b5d.
 2011-10-11 23:49:09,579 DEBUG 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Ignoring table not 
 disabled exception for supporting online schema changes.
 2011-10-11 23:49:09,580 INFO 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Handling table 
 operation C_M_MODIFY_TABLE on table TestTable
 2011-10-11 23:49:09,612 INFO org.apache.hadoop.hbase.util.FSUtils: 
 TableInfoPath = hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo tmpPath = 
 hdfs://sv4r11s38:9100/hbase/TestTable/.tmp/.tableinfo.1318376949612
 2011-10-11 23:49:09,692 INFO org.apache.hadoop.hbase.util.FSUtils: 
 TableDescriptor stored. TableInfoPath = 
 hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo
 2011-10-11 23:49:09,693 INFO org.apache.hadoop.hbase.util.FSUtils: Updated 
 tableinfo=hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo to blah
 2011-10-11 23:49:09,695 INFO 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Bucketing regions 
 by region server...
 2011-10-11 23:49:09,695 DEBUG org.apache.hadoop.hbase.client.MetaScanner: 
 Scanning .META. starting at row=TestTable,,00 for max=2147483647 
 rows
 2011-10-11 23:49:09,709 DEBUG 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
 The connection to hconnection-0x132f043bbde02e9 has been closed.
 2011-10-11 23:49:09,709 ERROR org.apache.hadoop.hbase.executor.EventHandler: 
 Caught throwable while processing event C_M_MODIFY_TABLE
 java.lang.NullPointerException
   at java.util.TreeMap.getEntry(TreeMap.java:324)
   at java.util.TreeMap.containsKey(TreeMap.java:209)
   at 
 org.apache.hadoop.hbase.master.handler.TableEventHandler.reOpenAllRegions(TableEventHandler.java:114)
   at 
 org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:90)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:168)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 {quote}
 The first time the shell reported that all the regions were updated 
 correctly, the second time it got stuck for a while:
 {quote}
 6/14 regions updated.
 0/14 regions updated.
 ...
 0/14 regions updated.
 2/16 regions updated.
 ...
 2/16 regions updated.
 8/9 regions updated.
 ...
 8/9 regions updated.
 {quote}
 After which I killed it, redid the alter and it worked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4649) Add atomic bulk load function to region server


[ 
https://issues.apache.org/jira/browse/HBASE-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134233#comment-13134233
 ] 

Ted Yu commented on HBASE-4649:
---

There are a few blank lines in TestHRegionServerBulkLoad.
There seems to be some overlap between the test and 
TestLoadIncrementalHFiles.java

Since the patch is part of fix for HBASE-4552, I wonder what the plan is to 
modify LoadIncrementalHFiles where the list of family, path pairs would be 
built by multiple threads. In other words, what data structure would be used to 
accommodate concurrency, assuming the current form of LoadIncrementalHFiles is 
kept ?

The above is related to my comment on HBASE-4552 @ 12/Oct/11 19:00.

 Add atomic bulk load function to region server
 --

 Key: HBASE-4649
 URL: https://issues.apache.org/jira/browse/HBASE-4649
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Affects Versions: 0.90.4, 0.92.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 
 0001-HBASE-4649-Add-atomic-bulk-load-function-to-region-s.patch


 Add a method that atomically bulk load multiple hfiles.  Row atomicity 
 guarantees for multi-column family rows require this functionality.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3992) Evaluate Lock Free Skip Tree

2011-10-24 Thread Nicolas Spiegelberg (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134234#comment-13134234
 ] 

stack commented on HBASE-3992:
--

There is also this that Ted turned up: https://github.com/nbronson/snaptree/

 Evaluate Lock Free Skip Tree
 

 Key: HBASE-3992
 URL: https://issues.apache.org/jira/browse/HBASE-3992
 Project: HBase
  Issue Type: Improvement
Reporter: Jason Rutherglen
Priority: Minor

 We can test out this variant of the ConcurrentSkipListMap.  
 Drop-in replacement for java.util.concurrent.ConcurrentSkipList[Map|Set]
 https://github.com/mspiegel/lockfreeskiptree

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4649) Add atomic bulk load function to region server

2011-10-24 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134243#comment-13134243
 ] 

Jonathan Hsieh commented on HBASE-4649:
---

@Ted

The existing LoadIncrementalHFiles emphasizes behavior when bulk imported 
HFiles have the wrong region boundaries and require split.  The new tests in 
this patch focuses on the the atomicity properties.  createHFile was borrowed 
and modified so that the expected values would easy to demonstrate atomicity 
failures.  More code was actually borrowed from TestAcidGuarentees.

I'll comment about LoadIncrementalHFiles in HBASE-4649 -- I'm in the process of 
cleaning up a first cut patch.

 Add atomic bulk load function to region server
 --

 Key: HBASE-4649
 URL: https://issues.apache.org/jira/browse/HBASE-4649
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Affects Versions: 0.90.4, 0.92.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 
 0001-HBASE-4649-Add-atomic-bulk-load-function-to-region-s.patch


 Add a method that atomically bulk load multiple hfiles.  Row atomicity 
 guarantees for multi-column family rows require this functionality.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4649) Add atomic bulk load function to region server

2011-10-24 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134244#comment-13134244
 ] 

Jonathan Hsieh commented on HBASE-4649:
---

I meant comment about LoadIncrementalHFiles in HBASE-4650

 Add atomic bulk load function to region server
 --

 Key: HBASE-4649
 URL: https://issues.apache.org/jira/browse/HBASE-4649
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Affects Versions: 0.90.4, 0.92.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 
 0001-HBASE-4649-Add-atomic-bulk-load-function-to-region-s.patch


 Add a method that atomically bulk load multiple hfiles.  Row atomicity 
 guarantees for multi-column family rows require this functionality.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4577) Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB

2011-10-24 Thread Jean-Daniel Cryans (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134252#comment-13134252
]

Jean-Daniel Cryans commented on HBASE-4577:
---

I'm not sure what the best solution is, but I think the current one is
confusing since both metrics share the same name with only Uncompressed being
different. Looking at the code it seems to be also confused.

The javadoc for Store.getStoreSizeUncompressed:
bq. The size of the store files, in bytes, uncompressed.

And for Store.getStorefilesSize:
bq. The size of the store files, in bytes.

Even the name of the variable is misleading because it's total data size, not
total size:
bq. totalUncompressedBytes

They read the same, but the first one is really the size of the uncompressed
data only. I think names should be changed to reflect what they really mean, no
need for new metrics.

Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
-

Key: HBASE-4577
URL: https://issues.apache.org/jira/browse/HBASE-4577
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: gaojinchao
Priority: Minor
Fix For: 0.92.0

Minor issue while looking at the RS metrics:
bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418,
storefileSizeMB=2420, compressionRatio=1.0008
I guess there's a truncation somewhere when it's adding the numbers up.
FWIW there's no compression on that table.

[jira] [Commented] (HBASE-4650) Update LoadIncrementalHFiles to use atomic bulk load RS mechanism

2011-10-24 Thread Jonathan Hsieh (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134253#comment-13134253
]

Jonathan Hsieh commented on HBASE-4650:
---

I'm in the process of cleaning up the modifications to LoadIncrementalHFiles
and adding more tests before submitting HBASE-4650 for a for review. This cut
passes the two tests that use LoadIncrementalHFiles (TestLoadIncrementalHFiles
and TestHFileOutputFormat). I'll post a preliminary version for those
interested.

In the code, there are significant logic changes due to grouping so I've chosen
to take out the concurrency on the first cut because gathering and splitting
HFiles into proper groups introduces a synchronization point that prevents some
of the concurrency as before. This is because groups need to be fully gathered
before bulk loads in a region is attempted. I'll include comments where
concurrency is ok.

Before I spend effort to parallelize this implementation more, I want to add
another test to verify that this works while splits are going on.

Update LoadIncrementalHFiles to use atomic bulk load RS mechanism
-

Key: HBASE-4650
URL: https://issues.apache.org/jira/browse/HBASE-4650
Project: HBase
Issue Type: Sub-task
Components: regionserver
Reporter: Jonathan Hsieh
Fix For: 0.92.0

MR jobs and command line bulk load execution runs use the
LoadIncrementalHFile.doBulkLoad. This needs to be updated to group HFiles by
row/region so that rows can be atomically loaded multiple column families.

[jira] [Commented] (HBASE-4649) Add atomic bulk load function to region server

2011-10-24 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134266#comment-13134266
 ] 

Jonathan Hsieh commented on HBASE-4649:
---

With respect to using a Queue vs List, I think he argument is moot -- because 
of the gather step we have a different instance of the region map, and 
different list of things to bulk load per iteration.

 Add atomic bulk load function to region server
 --

 Key: HBASE-4649
 URL: https://issues.apache.org/jira/browse/HBASE-4649
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Affects Versions: 0.90.4, 0.92.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: 
 0001-HBASE-4649-Add-atomic-bulk-load-function-to-region-s.patch


 Add a method that atomically bulk load multiple hfiles.  Row atomicity 
 guarantees for multi-column family rows require this functionality.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter


[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134277#comment-13134277
 ] 

Nicolas Spiegelberg commented on HBASE-4532:


+1 on commit.  TestHCM is an issue unrelated to this JIRA and shouldn't hold it 
up.  Should use 'git bisect' to figure out where it was introduced and comment 
on that JIRA.

 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4627) Ability to specify a custom start/end to RegionSplitter

2011-10-24 Thread Nicolas Spiegelberg (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg updated HBASE-4627:
---

Comment: was deleted

(was: nspiegelberg requested code review of [jira] [HBASE-4627] Ability to 
specify a custom start/end to RegionSplitter.
Reviewers: DUMMY_REVIEWER

  [HBASE-4627]

  added a custom start/end row to RegionSplitter.  Also solved
  an off-by-one error because the end row is prefix-inclusive and not
  exclusive.

  a href=https://issues.apache.org/jira/browse/HBASE-4489; title=Better key 
splitting in RegionSplitterdelHBASE-4489/del/a changed the default 
endKey on HexStringSplit from 7FFF... to ...  While this is correct, 
existing users of 0.90 RegionSplitter have 7FFF as the end key in their schema 
and the last region will not split properly under this new code.  We need to 
let the user specify a custom start/end key range for when situations like this 
arise.  Optimally, we should also write the start/end key in META so we could 
figure this out implicitly instead of requiring the user to explicitly specify 
it.

TEST PLAN
   - mvn test -Dtest=TestRegionSplitter

REVISION DETAIL
  http://reviews.facebook.net/D39

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/util/Bytes.java
  src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
  src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java

MANAGE HERALD DIFFERENTIAL RULES
  http://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  http://reviews.facebook.net/herald/transcript/75/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.
)

 Ability to specify a custom start/end to RegionSplitter
 ---

 Key: HBASE-4627
 URL: https://issues.apache.org/jira/browse/HBASE-4627
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
 Attachments: D39.1.patch, D39.1.patch


 HBASE-4489 changed the default endKey on HexStringSplit from 7FFF... to 
 ...  While this is correct, existing users of 0.90 RegionSplitter have 
 7FFF as the end key in their schema and the last region will not split 
 properly under this new code.  We need to let the user specify a custom 
 start/end key range for when situations like this arise.  Optimally, we 
 should also write the start/end key in META so we could figure this out 
 implicitly instead of requiring the user to explicitly specify it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4549) Add thrift API to read version and build date of HBase

2011-10-24 Thread Song Liu (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134285#comment-13134285
 ] 

Song Liu commented on HBASE-4549:
-

ping Jon

 Add thrift API to read version and build date of HBase 
 ---

 Key: HBASE-4549
 URL: https://issues.apache.org/jira/browse/HBASE-4549
 Project: HBase
  Issue Type: Improvement
  Components: thrift
Reporter: Song Liu
Priority: Minor
 Attachments: patch-hbase-4549.txt

   Original Estimate: 2h
  Remaining Estimate: 2h

 Adding API to get the hbase server version and build date will be helpful for 
 the client to communicate with different versions of the server accordingly. 
 class VersionInfo can be reused to provide required information. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4650) Update LoadIncrementalHFiles to use atomic bulk load RS mechanism

2011-10-24 Thread Karthik Ranganathan (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4650:
--

Attachment: 
0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.prelim.patch

Here's my preliminary non parallelized implementation.

 Update LoadIncrementalHFiles to use atomic bulk load RS mechanism
 -

 Key: HBASE-4650
 URL: https://issues.apache.org/jira/browse/HBASE-4650
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Reporter: Jonathan Hsieh
 Fix For: 0.92.0

 Attachments: 
 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.prelim.patch


 MR jobs and command line bulk load execution runs use the 
 LoadIncrementalHFile.doBulkLoad.  This needs to be updated to group HFiles by 
 row/region so that rows can be atomically loaded multiple column families.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4651) ConcurrentModificationException might be thrown in TestHCM.testConnectionUniqueness

2011-10-24 Thread Ted Yu (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4651:
--

Issue Type: Test  (was: Bug)

Integrated to 0.92 and TRUNK.

Thanks for the patch Jinchao.

 ConcurrentModificationException might be thrown in 
 TestHCM.testConnectionUniqueness
 ---

 Key: HBASE-4651
 URL: https://issues.apache.org/jira/browse/HBASE-4651
 Project: HBase
  Issue Type: Test
Affects Versions: 0.92.0
Reporter: gaojinchao
Assignee: gaojinchao
Priority: Minor
 Fix For: 0.92.0

 Attachments: HBASE-4651_Trunk.patch


 See
 https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2357/testReport/junit/org.apache.hadoop.hbase.client/TestHCM/testConnectionUniqueness
 363 EntryK,V nextEntry() {
 364 if (modCount != expectedModCount)
 365 throw new
 ConcurrentModificationExceptionhttp://kickjava.com/src/java/util/ConcurrentModificationException.java.htm
 [image:
 JavaDoc] http://kickjava.com/2487.htm();
 Read more:
 http://kickjava.com/src/java/util/LinkedHashMap.java.htm#ixzz1bbCC0gaT
 HCM uses proper synchronization when accessing HBASE_INSTANCES.
 Looking at TestHCM.getValidKeyCount(), it puts values of HBASE_INSTANCES in a 
 Set and returns the size of the Set.
 However, post HBASE-3777, the values (HConnectionImplementation's) in 
 HBASE_INSTANCES would be unique.
 TestHCM.getValidKeyCount() can be removed from the test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4651) ConcurrentModificationException might be thrown in TestHCM.testConnectionUniqueness

2011-10-24 Thread Ted Yu (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HBASE-4651.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

 ConcurrentModificationException might be thrown in 
 TestHCM.testConnectionUniqueness
 ---

 Key: HBASE-4651
 URL: https://issues.apache.org/jira/browse/HBASE-4651
 Project: HBase
  Issue Type: Test
Affects Versions: 0.92.0
Reporter: gaojinchao
Assignee: gaojinchao
Priority: Minor
 Fix For: 0.92.0

 Attachments: HBASE-4651_Trunk.patch


 See
 https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2357/testReport/junit/org.apache.hadoop.hbase.client/TestHCM/testConnectionUniqueness
 363 EntryK,V nextEntry() {
 364 if (modCount != expectedModCount)
 365 throw new
 ConcurrentModificationExceptionhttp://kickjava.com/src/java/util/ConcurrentModificationException.java.htm
 [image:
 JavaDoc] http://kickjava.com/2487.htm();
 Read more:
 http://kickjava.com/src/java/util/LinkedHashMap.java.htm#ixzz1bbCC0gaT
 HCM uses proper synchronization when accessing HBASE_INSTANCES.
 Looking at TestHCM.getValidKeyCount(), it puts values of HBASE_INSTANCES in a 
 Set and returns the size of the Set.
 However, post HBASE-3777, the values (HConnectionImplementation's) in 
 HBASE_INSTANCES would be unique.
 TestHCM.getValidKeyCount() can be removed from the test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4191) Utilize getTopBlockLocations in load balancer


[ 
https://issues.apache.org/jira/browse/HBASE-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134308#comment-13134308
 ] 

Karthik Ranganathan commented on HBASE-4191:


I think we should also go one step further and store the preferred assignments 
in a column-family in META. That way, upon restarts the locality is 
automatically preserved.

 Utilize getTopBlockLocations in load balancer
 -

 Key: HBASE-4191
 URL: https://issues.apache.org/jira/browse/HBASE-4191
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
Assignee: Liyin Tang

 HBASE-4114 implemented getTopBlockLocations().
 Load balancer should utilize this method and assign the region to be moved to 
 the region server with the highest block affinity.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4653) Master can't easily get rid of LZO compressed tables when the codec isn't available

2011-10-24 Thread Benoit Sigoure (Created) (JIRA)

Master can't easily get rid of LZO compressed tables when the codec isn't 
available
---

 Key: HBASE-4653
 URL: https://issues.apache.org/jira/browse/HBASE-4653
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.4
Reporter: Benoit Sigoure
Priority: Minor



I mistakenly created a table with {{COMPRESSION = LZO}}, and I don't have LZO 
installed.  I'm running a vanilla 0.90.4 release.  The master is unable to 
deploy the region of that table because the codec is missing.  I can't get rid 
of it.  I can't drop the table from  the shell, although it seems I could 
disable it.  Thankfully I found a workaround for this bug (see further below).

{code}
hbase(main):003:0 disable 'mytable'
0 row(s) in 1.1010 seconds

hbase(main):004:0 drop 'mytable'
[hung forever]
{code}

in the logs:

{code}
2011-10-22 03:05:42,153 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Instantiated mytable,,1319278131519.6eb6891a8b072402b5064f4cc68c210d.
2011-10-22 03:05:42,154 ERROR 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of 
region=mytable,,1319278131519.6eb6891a8b072402b5064f4cc68c210d. 
java.io.IOException: java.lang.RuntimeException:
java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec
   at 
org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:89)
   at 
org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:2573)
   at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2562)
   at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2550)
   at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:272)
   at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:99)
   at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:156)
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: 
com.hadoop.compression.lzo.LzoCodec
   at 
org.apache.hadoop.hbase.io.hfile.Compression$Algorithm$1.getCodec(Compression.java:92)
   at 
org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:197)
   at 
org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:84)
   ... 9 more
Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
   at 
org.apache.hadoop.hbase.io.hfile.Compression$Algorithm$1.getCodec(Compression.java:87)
   ... 11 more
[...]
2011-10-22 03:15:30,620 DEBUG 
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region to 
clear regions in transition; 
mytable,,1319278131519.6eb6891a8b072402b5064f4cc68c210d. state=OPENING, 
ts=1319278483001
2011-10-22 03:15:31,621 DEBUG 
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region to 
clear regions in transition;
mytable,,1319278131519.6eb6891a8b072402b5064f4cc68c210d. state=OPENING, 
ts=1319278483001
[repeat message above indefinitely every 1s]
{code}

I tried restarting HBase, no luck.  How do I get rid of this table so I can 
recreate it without {{COMPRESSION = LZO}}?

h2. Workaround

Change the schema for each family, restart HBase, drop the table.

{code}
hbase(main):004:0 alter 'mytable', {NAME = 'fam1', COMPRESSION = 'NONE'}
0 row(s) in 0.1160 seconds

hbase(main):005:0 alter 'mytable', {NAME = 'fam2', COMPRESSION = 'NONE'}
0 row(s) in 0.0480 seconds

hbase(main):007:0 drop 'mytable'
^C
[hung forever]
{code}

[restart HBase]  :(

{code}
hbase(main):001:0 disable 'mytable'
0 row(s) in 2.5010 seconds

hbase(main):002:0 drop 'mytable'
0 row(s) in 1.1240 seconds
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4653) Master can't easily get rid of LZO compressed tables when the codec isn't available

2011-10-24 Thread Lars Hofhansl (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134334#comment-13134334
 ] 

Lars Hofhansl commented on HBASE-4653:
--

Yep... I've found that too. Didn't think to file a case, though. :)
Maybe the answer is to disallow creation of CFs with unsupported compression.


 Master can't easily get rid of LZO compressed tables when the codec isn't 
 available
 ---

 Key: HBASE-4653
 URL: https://issues.apache.org/jira/browse/HBASE-4653
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.90.4
Reporter: Benoit Sigoure
Priority: Minor

 I mistakenly created a table with {{COMPRESSION = LZO}}, and I don't have 
 LZO installed.  I'm running a vanilla 0.90.4 release.  The master is unable 
 to deploy the region of that table because the codec is missing.  I can't get 
 rid of it.  I can't drop the table from  the shell, although it seems I could 
 disable it.  Thankfully I found a workaround for this bug (see further below).
 {code}
 hbase(main):003:0 disable 'mytable'
 0 row(s) in 1.1010 seconds
 hbase(main):004:0 drop 'mytable'
 [hung forever]
 {code}
 in the logs:
 {code}
 2011-10-22 03:05:42,153 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
 Instantiated mytable,,1319278131519.6eb6891a8b072402b5064f4cc68c210d.
 2011-10-22 03:05:42,154 ERROR 
 org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open 
 of region=mytable,,1319278131519.6eb6891a8b072402b5064f4cc68c210d. 
 java.io.IOException: java.lang.RuntimeException:
 java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec
at 
 org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:89)
at 
 org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:2573)
at 
 org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2562)
at 
 org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2550)
at 
 org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:272)
at 
 org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:99)
at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:156)
at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
 Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: 
 com.hadoop.compression.lzo.LzoCodec
at 
 org.apache.hadoop.hbase.io.hfile.Compression$Algorithm$1.getCodec(Compression.java:92)
at 
 org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:197)
at 
 org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:84)
... 9 more
 Caused by: java.lang.ClassNotFoundException: 
 com.hadoop.compression.lzo.LzoCodec
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at 
 org.apache.hadoop.hbase.io.hfile.Compression$Algorithm$1.getCodec(Compression.java:87)
... 11 more
 [...]
 2011-10-22 03:15:30,620 DEBUG 
 org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region 
 to clear regions in transition; 
 mytable,,1319278131519.6eb6891a8b072402b5064f4cc68c210d. state=OPENING, 
 ts=1319278483001
 2011-10-22 03:15:31,621 DEBUG 
 org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region 
 to clear regions in transition;
 mytable,,1319278131519.6eb6891a8b072402b5064f4cc68c210d. state=OPENING, 
 ts=1319278483001
 [repeat message above indefinitely every 1s]
 {code}
 I tried restarting HBase, no luck.  How do I get rid of this table so I can 
 recreate it without {{COMPRESSION = LZO}}?
 h2. Workaround
 Change the schema for each family, restart HBase, drop the table.
 {code}
 hbase(main):004:0 alter 'mytable', {NAME = 'fam1', COMPRESSION = 'NONE'}
 0 row(s) in 0.1160 seconds
 hbase(main):005:0 alter 'mytable', {NAME = 'fam2', COMPRESSION = 'NONE'}
 0 row(s) in 0.0480 seconds
 hbase(main):007:0 drop 'mytable'
 ^C
 [hung forever]
 {code}
 [restart HBase]  :(
 {code}
 hbase(main):001:0 disable 'mytable'
 0 row(s) in 2.5010 seconds
 hbase(main):002:0 drop 'mytable'
 0 row(s) in

[jira] [Commented] (HBASE-4648) Bytes.toBigDecimal() doesn't use offset

2011-10-24 Thread Lars Hofhansl (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134336#comment-13134336
 ] 

Lars Hofhansl commented on HBASE-4648:
--

Whoa... Good find Bryan.

 Bytes.toBigDecimal() doesn't use offset
 ---

 Key: HBASE-4648
 URL: https://issues.apache.org/jira/browse/HBASE-4648
 Project: HBase
  Issue Type: Bug
  Components: util
Affects Versions: 0.90.4
 Environment: Java 1.6.0_26, Mac OS X 10.7 and CentOS 6
Reporter: Bryan Keller

 The Bytes.toBigDecimal(byte[], offset, len) method does not use the offset, 
 thus you will get an incorrect result for the BigDecimal unless the 
 BigDecimal's bytes are at the beginning of the byte array.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4224) Need a flush by regionserver rather than by table option


[ 
https://issues.apache.org/jira/browse/HBASE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134355#comment-13134355
 ] 

Ted Yu commented on HBASE-4224:
---

Overall, patch looks good.

A better name for getRegionServerPairs() would be 
getRegionServerPairsForTable().
The LOG.info() in isValidRegionServerPair() should be LOG.debug(). Also, 
offline region isn't covered in the log message.

There are some formatting improvements, such as:
{code}
if(flush(serverName, pair.getFirst())) {
{code}
There should be a space between if and (.

Please also add support for this feature to hbase shell.


 Need a flush by regionserver rather than by table option
 

 Key: HBASE-4224
 URL: https://issues.apache.org/jira/browse/HBASE-4224
 Project: HBase
  Issue Type: Bug
  Components: shell
Reporter: stack
Assignee: Akash Ashok
 Attachments: HBase-4224.patch


 This evening needed to clean out logs on the cluster.  logs are by 
 regionserver.  to let go of logs, we need to have all edits emptied from 
 memory.  only flush is by table or region.  We need to be able to flush the 
 regionserver.  Need to add this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4629) enable automated patch testing for hbase


[ 
https://issues.apache.org/jira/browse/HBASE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134363#comment-13134363
 ] 

stack commented on HBASE-4629:
--

@Giri Are we supposed to commit this to hbase?  It creates a dir named 
dev-support at top level in hbase.  Is that what you want?

 enable automated patch testing for hbase
 

 Key: HBASE-4629
 URL: https://issues.apache.org/jira/browse/HBASE-4629
 Project: HBase
  Issue Type: New Feature
Reporter: Giridharan Kesavan
Assignee: Giridharan Kesavan
 Attachments: HBASE-4629.patch


 enable jenkins automated patch testing for hbase project

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4648) Bytes.toBigDecimal() doesn't use offset

2011-10-24 Thread Bryan Keller (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134386#comment-13134386
 ] 

Bryan Keller commented on HBASE-4648:
-

I'll upload the patch a little later today, sorry for the delay.

 Bytes.toBigDecimal() doesn't use offset
 ---

 Key: HBASE-4648
 URL: https://issues.apache.org/jira/browse/HBASE-4648
 Project: HBase
  Issue Type: Bug
  Components: util
Affects Versions: 0.90.4
 Environment: Java 1.6.0_26, Mac OS X 10.7 and CentOS 6
Reporter: Bryan Keller

 The Bytes.toBigDecimal(byte[], offset, len) method does not use the offset, 
 thus you will get an incorrect result for the BigDecimal unless the 
 BigDecimal's bytes are at the beginning of the byte array.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4634) test.build.data property overused leading to write data at the wrong place

2011-10-24 Thread nkeywal (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4634:
---

Attachment: 20111024_4634_all.v5.patch

v5, all tests ok, after a refresh.

 test.build.data property overused leading to write data at the wrong place
 

 Key: HBASE-4634
 URL: https://issues.apache.org/jira/browse/HBASE-4634
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.92.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
 Attachments: 2011020_4634_all.patch, 
 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch, 
 20111022_4634_all.v3.patch, 20111023_4634_all.v4.patch, 
 20111024_4634_all.v5.patch


 test.build.data is overloaded in HBase.At the beginning, it's the Default 
 parent directory for test output., but then it's rewritten to be the 
 directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
 It seems that this value is already used by MiniDFS (i.e. outside of HBase): 
 Name is as it is because mini dfs has hard-codings to put test data here.
 As it is today, there is at least a bug in HBaseTestingUtility:
 {noformat}
   public void initTestDir() {
 if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
   clusterTestBuildDir = setupClusterTestBuildDir();
   System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
 }
   }
 {noformat}
 if you set a value for test.build.data, the test dir will be the parent 
 directory and not a temp subdir, leading to issues as multiple tests will 
 end-ups in the same (bad) directory. This function is barely used today, 
 hence it's not visible, but I would like to use it in some new code.
 A possible fix is to remove the check for null and continue with the 
 overloading, but I don't think it would be a big issue to create a new 
 key(like test.build.data.rootdirectory) specific to the root directory and 
 to use test.build.data only to communicate with MiniDFS. Feedback welcome.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4577) Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB

2011-10-24 Thread Matt Corgan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134408#comment-13134408
 ] 

Matt Corgan commented on HBASE-4577:


I think the intention of adding it was to see how big the data would be in 
memory as opposed to on disk, which is a valuable metric.  However, we're 
already jumping ahead to doing delta encoding and prefix compression, so there 
will soon be a need for a third metric to track encoded size.  Maybe these 3 
names would be better:

storefileSize: size as reported by the filesystem (lzo/gzip compressed)

encodedDataSize: size in the block cache (with delta encoding or prefix 
compression, but no gzip)

rawDataSize (instead of uncompressedBytes): size when stored in the current 
concatenated KeyValue format (the biggest of the 3)

The last 2 would only count datablocks of KeyValues.  I'm not sure where 
bloomfilters and indexblocks should be counted into these.  Possibly separate 
metrics?

 Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
 -

 Key: HBASE-4577
 URL: https://issues.apache.org/jira/browse/HBASE-4577
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: gaojinchao
Priority: Minor
 Fix For: 0.92.0


 Minor issue while looking at the RS metrics:
 bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, 
 storefileSizeMB=2420, compressionRatio=1.0008
 I guess there's a truncation somewhere when it's adding the numbers up.
 FWIW there's no compression on that table.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4629) enable automated patch testing for hbase

2011-10-24 Thread Giridharan Kesavan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134418#comment-13134418
 ] 

Giridharan Kesavan commented on HBASE-4629:
---

@Stack. Please commit this patch to hbase.

This patch would create a dir - dev-support with two files test-patch.sh and 
test-patch.properties. Instead of having test-patch script as part of the 
source code it's better to have it at the top level in a diff directory.

 enable automated patch testing for hbase
 

 Key: HBASE-4629
 URL: https://issues.apache.org/jira/browse/HBASE-4629
 Project: HBase
  Issue Type: New Feature
Reporter: Giridharan Kesavan
Assignee: Giridharan Kesavan
 Attachments: HBASE-4629.patch


 enable jenkins automated patch testing for hbase project

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter


[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134421#comment-13134421
 ] 

Hudson commented on HBASE-4532:
---

Integrated in HBase-TRUNK #2363 (See 
[https://builds.apache.org/job/HBase-TRUNK/2363/])
HBASE-4532 Avoid top row seek by dedicated bloom filter for delete family 
bloom filter

nspiegelberg : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java


 Avoid top row seek by dedicated bloom filter for delete family bloom filter
 ---

 Key: HBASE-4532
 URL: https://issues.apache.org/jira/browse/HBASE-4532
 Project: HBase
  Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
 Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
 hbase-4532-89-fb.patch


 The previous jira, HBASE-4469, is to avoid the top row seek operation if 
 row-col bloom filter is enabled. 
 This jira tries to avoid top row seek for all the cases by creating a 
 dedicated bloom filter only for delete family
 The only subtle use case is when we are interested in the top row with empty 
 column.
 For example, 
 we are interested in row1/cf1:/1/put.
 So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
 bloom filter will say there is NO delete family.
 Then it will avoid the top row seek and return a fake kv, which is the last 
 kv for this row (createLastOnRowCol).
 In this way, we have already missed the real kv we are interested in.
 The solution for the above problem is to disable this optimization if we are 
 trying to GET/SCAN a row with empty column.
 Evaluation from TestSeekOptimization:
 Previously:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1714 (68.40%), savings: 31.60%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
 enabled.[HBASE-4469]
 
 After this change:
 For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
 optimization: 1458 (58.18%), savings: 41.82%
 So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please

[jira] [Commented] (HBASE-4651) ConcurrentModificationException might be thrown in TestHCM.testConnectionUniqueness


[ 
https://issues.apache.org/jira/browse/HBASE-4651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134420#comment-13134420
 ] 

Hudson commented on HBASE-4651:
---

Integrated in HBase-TRUNK #2363 (See 
[https://builds.apache.org/job/HBase-TRUNK/2363/])
HBASE-4651  ConcurrentModificationException might be thrown in
   TestHCM.testConnectionUniqueness (Jinchao)

tedyu : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java


 ConcurrentModificationException might be thrown in 
 TestHCM.testConnectionUniqueness
 ---

 Key: HBASE-4651
 URL: https://issues.apache.org/jira/browse/HBASE-4651
 Project: HBase
  Issue Type: Test
Affects Versions: 0.92.0
Reporter: gaojinchao
Assignee: gaojinchao
Priority: Minor
 Fix For: 0.92.0

 Attachments: HBASE-4651_Trunk.patch


 See
 https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2357/testReport/junit/org.apache.hadoop.hbase.client/TestHCM/testConnectionUniqueness
 363 EntryK,V nextEntry() {
 364 if (modCount != expectedModCount)
 365 throw new
 ConcurrentModificationExceptionhttp://kickjava.com/src/java/util/ConcurrentModificationException.java.htm
 [image:
 JavaDoc] http://kickjava.com/2487.htm();
 Read more:
 http://kickjava.com/src/java/util/LinkedHashMap.java.htm#ixzz1bbCC0gaT
 HCM uses proper synchronization when accessing HBASE_INSTANCES.
 Looking at TestHCM.getValidKeyCount(), it puts values of HBASE_INSTANCES in a 
 Set and returns the size of the Set.
 However, post HBASE-3777, the values (HConnectionImplementation's) in 
 HBASE_INSTANCES would be unique.
 TestHCM.getValidKeyCount() can be removed from the test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4629) enable automated patch testing for hbase


 [ 
https://issues.apache.org/jira/browse/HBASE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4629.
--

   Resolution: Fixed
Fix Version/s: 0.94.0
 Hadoop Flags: Reviewed

Committed to TRUNK.  Thanks for the patch Giri.

 enable automated patch testing for hbase
 

 Key: HBASE-4629
 URL: https://issues.apache.org/jira/browse/HBASE-4629
 Project: HBase
  Issue Type: New Feature
Reporter: Giridharan Kesavan
Assignee: Giridharan Kesavan
 Fix For: 0.94.0

 Attachments: HBASE-4629.patch


 enable jenkins automated patch testing for hbase project

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4654) [replication] Add a check to make sure we don't replicate to ourselves

2011-10-24 Thread Jean-Daniel Cryans (Created) (JIRA)

[replication] Add a check to make sure we don't replicate to ourselves
--

 Key: HBASE-4654
 URL: https://issues.apache.org/jira/browse/HBASE-4654
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
 Fix For: 0.90.5


It's currently possible to add a peer for replication and point it to the local 
cluster, which I believe could very well happen for those like us that use only 
one ZK ensemble per DC so that only the root znode changes when you want to set 
up replication intra-DC.

I don't think comparing just the cluster ID would be enough because you would 
normally use a different one for another cluster and nothing will block you 
from pointing elsewhere.
Comparing the ZK ensemble address doesn't work either when you have multiple 
DNS entries that point at the same place.

I think this could be resolved by looking up the master address in the relevant 
znode as it should be exactly the same thing in the case where you have the 
same cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4191) Utilize getTopBlockLocations in load balancer


[ 
https://issues.apache.org/jira/browse/HBASE-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134457#comment-13134457
 ] 

Ted Yu commented on HBASE-4191:
---

Currently assignAllUserRegions() reads hbase.master.startup.retainassign and 
if the value is true, balancer.retainAssignment() is called.
Looks like new config parameter should be introduce, e.g. 
hbase.master.startup.policy, to accommodate the above and future enhancements.

 Utilize getTopBlockLocations in load balancer
 -

 Key: HBASE-4191
 URL: https://issues.apache.org/jira/browse/HBASE-4191
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
Assignee: Liyin Tang

 HBASE-4114 implemented getTopBlockLocations().
 Load balancer should utilize this method and assign the region to be moved to 
 the region server with the highest block affinity.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4591) TTL for old HLogs should be calculated from last modification time.

2011-10-24 Thread Nicolas Spiegelberg (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg updated HBASE-4591:
---

Attachment: 0007-HBASE-4591-Increase-TTL-for-old-HLogs-and-change-TTL.patch

patch generated from our internal branch, but applies cleanly to all open 
source branches with 'patch -p1'

 TTL for old HLogs should be calculated from last modification time.
 ---

 Key: HBASE-4591
 URL: https://issues.apache.org/jira/browse/HBASE-4591
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.89.20100621
Reporter: Madhuwanti Vaidya
Assignee: Madhuwanti Vaidya
Priority: Minor
 Attachments: 
 0007-HBASE-4591-Increase-TTL-for-old-HLogs-and-change-TTL.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4591) TTL for old HLogs should be calculated from last modification time.

2011-10-24 Thread Nicolas Spiegelberg (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134478#comment-13134478
 ] 

Ted Yu commented on HBASE-4591:
---

{code}
+} catch (IOException e) {
+  LOG.error(Unable to get modification time of file  + 
filePath.getName() +
{code}
Only the final component of this path is logged.
Should the full path be logged ?

 TTL for old HLogs should be calculated from last modification time.
 ---

 Key: HBASE-4591
 URL: https://issues.apache.org/jira/browse/HBASE-4591
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.89.20100621
Reporter: Madhuwanti Vaidya
Assignee: Madhuwanti Vaidya
Priority: Minor
 Attachments: 
 0007-HBASE-4591-Increase-TTL-for-old-HLogs-and-change-TTL.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4591) TTL for old HLogs should be calculated from last modification time.


[ 
https://issues.apache.org/jira/browse/HBASE-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134495#comment-13134495
 ] 

Nicolas Spiegelberg commented on HBASE-4591:


can change that to toString() to print the full pathname.  thanks Ted

 TTL for old HLogs should be calculated from last modification time.
 ---

 Key: HBASE-4591
 URL: https://issues.apache.org/jira/browse/HBASE-4591
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.89.20100621
Reporter: Madhuwanti Vaidya
Assignee: Madhuwanti Vaidya
Priority: Minor
 Attachments: 
 0007-HBASE-4591-Increase-TTL-for-old-HLogs-and-change-TTL.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4191) hbase load balancer needs locality awareness

2011-10-24 Thread Liyin Tang (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Liyin Tang updated HBASE-4191:
--

Description:
Previously, HBASE-4114 implements the metrics for HFile HDFS block locality,
which provides the HFile level locality information.
But in order to work with load balancer and region assignment, we need the
region level locality information.

Let's define the region locality information first, which is almost the same as
HFile locality index.

HRegion locality index (HRegion A, RegionServer B) =
(Total number of HDFS blocks that can be retrieved locally by the RegionServer
B for the HRegion A) / ( Total number of the HDFS blocks for the Region A)
So the HRegion locality index tells us that how much locality we can get if the
HMaster assign the HRegion A to the RegionServer B.

So there will be 2 steps involved to assign regions based on the locality.
1) During the cluster start up time, the master will scan the hdfs to calculate
the HRegion locality index for each pair of HRegion and Region Server. It is
pretty expensive to scan the dfs. So we only needs to do this once during the
start up time.

2) During the cluster run time, each region server will update the HRegion
locality index as metrics periodically as HBASE-4114 did. The Region Server
can expose them to the Master through ZK, meta table, or just RPC messages.

Based on the HRegion locality index, the assignment manager in the master
would have a global knowledge about the region locality distribution. Imaging
the HRegion locality index as the capacity between the region server set and
region set, the assignment manager could the run the MAXIMUM FLOW solver to
reach the global optimization.

In addition, the HBASE-4491 (Locality Checker) is the tool, which is based on
the same metrics, to proactively to scan dfs to calculate the global locality
information in the cluster. It will help us to verify data locality information
during the run time.

was:
HBASE-4114 implemented getTopBlockLocations().
Load balancer should utilize this method and assign the region to be moved to
the region server with the highest block affinity.

Issue Type: New Feature (was: Improvement)
Summary: hbase load balancer needs locality awareness (was: Utilize
getTopBlockLocations in load balancer)

hbase load balancer needs locality awareness

Key: HBASE-4191
URL: https://issues.apache.org/jira/browse/HBASE-4191
Project: HBase
Issue Type: New Feature
Reporter: Ted Yu
Assignee: Liyin Tang

Previously, HBASE-4114 implements the metrics for HFile HDFS block locality,
which provides the HFile level locality information.
But in order to work with load balancer and region assignment, we need the
region level locality information.
Let's define the region locality information first, which is almost the same
as HFile locality index.
HRegion locality index (HRegion A, RegionServer B) =
(Total number of HDFS blocks that can be retrieved locally by the
RegionServer B for the HRegion A) / ( Total number of the HDFS blocks for the
Region A)
So the HRegion locality index tells us that how much locality we can get if
the HMaster assign the HRegion A to the RegionServer B.
So there will be 2 steps involved to assign regions based on the locality.
1) During the cluster start up time, the master will scan the hdfs to
calculate the HRegion locality index for each pair of HRegion and Region
Server. It is pretty expensive to scan the dfs. So we only needs to do this
once during the start up time.
2) During the cluster run time, each region server will update the HRegion
locality index as metrics periodically as HBASE-4114 did. The Region Server
can expose them to the Master through ZK, meta table, or just RPC messages.
Based on the HRegion locality index, the assignment manager in the master
would have a global knowledge about the region locality distribution. Imaging
the HRegion locality index as the capacity between the region server set
and region set, the assignment manager could the run the MAXIMUM FLOW solver
to reach the global optimization.
In addition, the HBASE-4491 (Locality Checker) is the tool, which is based on
the same metrics, to proactively to scan dfs to calculate the global locality
information in the cluster. It will help us to verify data locality
information during the run time.

[jira] [Commented] (HBASE-4591) TTL for old HLogs should be calculated from last modification time.

2011-10-24 Thread Nicolas Spiegelberg (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134496#comment-13134496
 ] 

Nicolas Spiegelberg commented on HBASE-4591:


received +1s internally from karthik.ranga  kannanm.  Will commit

 TTL for old HLogs should be calculated from last modification time.
 ---

 Key: HBASE-4591
 URL: https://issues.apache.org/jira/browse/HBASE-4591
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.89.20100621
Reporter: Madhuwanti Vaidya
Assignee: Madhuwanti Vaidya
Priority: Minor
 Attachments: 
 0007-HBASE-4591-Increase-TTL-for-old-HLogs-and-change-TTL.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-1294) Add automated patch testing to issue tracker.

2011-10-24 Thread Nicolas Spiegelberg (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-1294.
--

Resolution: Duplicate

Marking as duplicate of HBASE-1294

 Add automated patch testing to issue tracker.
 -

 Key: HBASE-1294
 URL: https://issues.apache.org/jira/browse/HBASE-1294
 Project: HBase
  Issue Type: Improvement
  Components: build
Reporter: Nitay Joffe

 Hadoop has a system which automatically runs every submitted patch through a 
 slew of checks (e.g. ant test, checkstyle, findbugs, javadoc) which each 
 return +1/-1 on the patch. We should add a similar automated system to our 
 issue tracker.
 The ZooKeeper folks recently did this. Here's some info on their progress:
 https://issues.apache.org/jira/browse/ZOOKEEPER-349
 http://hudson.zones.apache.org/hudson/view/ZooKeeper/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4591) TTL for old HLogs should be calculated from last modification time.

2011-10-24 Thread Nicolas Spiegelberg (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg resolved HBASE-4591.


   Resolution: Fixed
Fix Version/s: 0.94.0
   0.92.0

 TTL for old HLogs should be calculated from last modification time.
 ---

 Key: HBASE-4591
 URL: https://issues.apache.org/jira/browse/HBASE-4591
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.89.20100621
Reporter: Madhuwanti Vaidya
Assignee: Madhuwanti Vaidya
Priority: Minor
 Fix For: 0.92.0, 0.94.0

 Attachments: 
 0007-HBASE-4591-Increase-TTL-for-old-HLogs-and-change-TTL.patch, 
 HBASE-4591-2.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4655) Document architecture of backups

2011-10-24 Thread Karthik Ranganathan (Created) (JIRA)

Document architecture of backups


 Key: HBASE-4655
 URL: https://issues.apache.org/jira/browse/HBASE-4655
 Project: HBase
  Issue Type: Sub-task
Reporter: Karthik Ranganathan
Assignee: Karthik Ranganathan


Basic idea behind the backup architecture for HBase

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4591) TTL for old HLogs should be calculated from last modification time.


[ 
https://issues.apache.org/jira/browse/HBASE-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134522#comment-13134522
 ] 

Nicolas Spiegelberg commented on HBASE-4591:


Breaks TestLogsCleaner.  Will come out with a test fix patch shortly.

 TTL for old HLogs should be calculated from last modification time.
 ---

 Key: HBASE-4591
 URL: https://issues.apache.org/jira/browse/HBASE-4591
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.89.20100621
Reporter: Madhuwanti Vaidya
Assignee: Madhuwanti Vaidya
Priority: Minor
 Fix For: 0.92.0, 0.94.0

 Attachments: 
 0007-HBASE-4591-Increase-TTL-for-old-HLogs-and-change-TTL.patch, 
 HBASE-4591-2.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4437) Update hadoop in 0.92 (0.20.205?)


 [ 
https://issues.apache.org/jira/browse/HBASE-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4437:
-

Attachment: 4437.txt

Move to 0.20.205.0 hadoop.  Also includes edits to our jsp and jamon templates 
commenting out DOCTYPE to get around bug where css is served as text/html.  See 
tail of HBASE-2110 for discussion.

 Update hadoop in 0.92 (0.20.205?)
 -

 Key: HBASE-4437
 URL: https://issues.apache.org/jira/browse/HBASE-4437
 Project: HBase
  Issue Type: Task
Reporter: stack
 Attachments: 4437.txt


 We ship with branch-0.20-append a few versions back from the tip.  If 205 
 comes out and hbase works on it, we should ship 0.92 with it (while also 
 ensuring it work w/ 0.22 and 0.23 branches).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4656) Note how dfs.support.append has to be enabled in 0.20.205.0 clusters

2011-10-24 Thread stack (Created) (JIRA)

Note how dfs.support.append has to be enabled in 0.20.205.0 clusters


 Key: HBASE-4656
 URL: https://issues.apache.org/jira/browse/HBASE-4656
 Project: HBase
  Issue Type: Task
Reporter: stack




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4437) Update hadoop in 0.92 (0.20.205?)


 [ 
https://issues.apache.org/jira/browse/HBASE-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4437.
--

   Resolution: Fixed
Fix Version/s: 0.92.0
 Assignee: stack
 Release Note: We ship with hadoop 0.20.205.0 now.  Make sure you have 
dfs.support.append enabled everywhere on your cluster.
 Hadoop Flags: Reviewed

 Update hadoop in 0.92 (0.20.205?)
 -

 Key: HBASE-4437
 URL: https://issues.apache.org/jira/browse/HBASE-4437
 Project: HBase
  Issue Type: Task
Reporter: stack
Assignee: stack
 Fix For: 0.92.0

 Attachments: 4437.txt


 We ship with branch-0.20-append a few versions back from the tip.  If 205 
 comes out and hbase works on it, we should ship 0.92 with it (while also 
 ensuring it work w/ 0.22 and 0.23 branches).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4591) TTL for old HLogs should be calculated from last modification time.


[ 
https://issues.apache.org/jira/browse/HBASE-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134545#comment-13134545
 ] 

Ted Yu commented on HBASE-4591:
---

In HBASE-4591-2.patch, I still see filePath.getName()
Do you mind changing that in the addendum ?

 TTL for old HLogs should be calculated from last modification time.
 ---

 Key: HBASE-4591
 URL: https://issues.apache.org/jira/browse/HBASE-4591
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.89.20100621
Reporter: Madhuwanti Vaidya
Assignee: Madhuwanti Vaidya
Priority: Minor
 Fix For: 0.92.0, 0.94.0

 Attachments: 
 0007-HBASE-4591-Increase-TTL-for-old-HLogs-and-change-TTL.patch, 
 HBASE-4591-2.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2611) Handle RS that fails while processing the failure of another one

2011-10-24 Thread Chris Trezzo (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134549#comment-13134549
 ] 

Chris Trezzo commented on HBASE-2611:
-

@J-D

If you don't mind, I was thinking about taking a crack at this using your 4 
types of znode strategy. I'll start working on a sketch patch.

At a first glance, it seems as though most of the code changes are going to be 
in ReplicationSourceManager.NodeFailoverWorker.run().

 Handle RS that fails while processing the failure of another one
 

 Key: HBASE-2611
 URL: https://issues.apache.org/jira/browse/HBASE-2611
 Project: HBase
  Issue Type: Sub-task
  Components: replication
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans

 HBASE-2223 doesn't manage region servers that fail while doing the transfer 
 of HLogs queues from other region servers that failed. Devise a reliable way 
 to do it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4650) Update LoadIncrementalHFiles to use atomic bulk load RS mechanism

2011-10-24 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134553#comment-13134553
 ] 

Jonathan Hsieh commented on HBASE-4650:
---

initial parallel implementation here: https://reviews.apache.org/r/2557/

 Update LoadIncrementalHFiles to use atomic bulk load RS mechanism
 -

 Key: HBASE-4650
 URL: https://issues.apache.org/jira/browse/HBASE-4650
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Reporter: Jonathan Hsieh
 Fix For: 0.92.0

 Attachments: 
 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.patch, 
 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.prelim.patch


 MR jobs and command line bulk load execution runs use the 
 LoadIncrementalHFile.doBulkLoad.  This needs to be updated to group HFiles by 
 row/region so that rows can be atomically loaded multiple column families.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4650) Update LoadIncrementalHFiles to use atomic bulk load RS mechanism


 [ 
https://issues.apache.org/jira/browse/HBASE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4650:
--

Attachment: 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.patch

 Update LoadIncrementalHFiles to use atomic bulk load RS mechanism
 -

 Key: HBASE-4650
 URL: https://issues.apache.org/jira/browse/HBASE-4650
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Reporter: Jonathan Hsieh
 Fix For: 0.92.0

 Attachments: 
 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.patch, 
 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.prelim.patch


 MR jobs and command line bulk load execution runs use the 
 LoadIncrementalHFile.doBulkLoad.  This needs to be updated to group HFiles by 
 row/region so that rows can be atomically loaded multiple column families.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4650) Update LoadIncrementalHFiles to use atomic bulk load RS mechanism


 [ 
https://issues.apache.org/jira/browse/HBASE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-4650:
--

Status: Patch Available  (was: Open)

 Update LoadIncrementalHFiles to use atomic bulk load RS mechanism
 -

 Key: HBASE-4650
 URL: https://issues.apache.org/jira/browse/HBASE-4650
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Reporter: Jonathan Hsieh
 Fix For: 0.92.0

 Attachments: 
 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.patch, 
 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.prelim.patch


 MR jobs and command line bulk load execution runs use the 
 LoadIncrementalHFile.doBulkLoad.  This needs to be updated to group HFiles by 
 row/region so that rows can be atomically loaded multiple column families.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4650) Update LoadIncrementalHFiles to use atomic bulk load RS mechanism

2011-10-24 Thread Jonathan Hsieh (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh reassigned HBASE-4650:
-

Assignee: Jonathan Hsieh

 Update LoadIncrementalHFiles to use atomic bulk load RS mechanism
 -

 Key: HBASE-4650
 URL: https://issues.apache.org/jira/browse/HBASE-4650
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.92.0

 Attachments: 
 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.patch, 
 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.prelim.patch


 MR jobs and command line bulk load execution runs use the 
 LoadIncrementalHFile.doBulkLoad.  This needs to be updated to group HFiles by 
 row/region so that rows can be atomically loaded multiple column families.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4629) enable automated patch testing for hbase


[ 
https://issues.apache.org/jira/browse/HBASE-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134555#comment-13134555
 ] 

Hudson commented on HBASE-4629:
---

Integrated in HBase-TRUNK #2364 (See 
[https://builds.apache.org/job/HBase-TRUNK/2364/])
HBASE-4629 enable automated patch testing for hbase

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/dev-support
* /hbase/trunk/dev-support/test-patch.properties
* /hbase/trunk/dev-support/test-patch.sh


 enable automated patch testing for hbase
 

 Key: HBASE-4629
 URL: https://issues.apache.org/jira/browse/HBASE-4629
 Project: HBase
  Issue Type: New Feature
Reporter: Giridharan Kesavan
Assignee: Giridharan Kesavan
 Fix For: 0.94.0

 Attachments: HBASE-4629.patch


 enable jenkins automated patch testing for hbase project

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2611) Handle RS that fails while processing the failure of another one

2011-10-24 Thread Chris Trezzo (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134568#comment-13134568
 ] 

Chris Trezzo commented on HBASE-2611:
-

...and of course ReplicationZookeeper.

 Handle RS that fails while processing the failure of another one
 

 Key: HBASE-2611
 URL: https://issues.apache.org/jira/browse/HBASE-2611
 Project: HBase
  Issue Type: Sub-task
  Components: replication
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans

 HBASE-2223 doesn't manage region servers that fail while doing the transfer 
 of HLogs queues from other region servers that failed. Devise a reliable way 
 to do it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2110) Move to ivy broke our finding stylesheet (Tables don't have blue lines around them any more).


[ 
https://issues.apache.org/jira/browse/HBASE-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134601#comment-13134601
 ] 

stack commented on HBASE-2110:
--

I commented out DOCTYPE for now in 0.92 branch and in TRUNK.

 Move to ivy broke our finding stylesheet (Tables don't have blue lines around 
 them any more).
 -

 Key: HBASE-2110
 URL: https://issues.apache.org/jira/browse/HBASE-2110
 Project: HBase
  Issue Type: Bug
Reporter: stack
  Labels: noob

 Move to ivy broke pulling in hbase.css from the static webapp; investigate.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4656) Note how dfs.support.append has to be enabled in 0.20.205.0 clusters


 [ 
https://issues.apache.org/jira/browse/HBASE-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4656:
-

Attachment: 4656.txt

Added note around our hadoop section to book and set dfs.support.append to true 
in hbase-default.xml

 Note how dfs.support.append has to be enabled in 0.20.205.0 clusters
 

 Key: HBASE-4656
 URL: https://issues.apache.org/jira/browse/HBASE-4656
 Project: HBase
  Issue Type: Task
Reporter: stack
 Fix For: 0.92.0

 Attachments: 4656.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4656) Note how dfs.support.append has to be enabled in 0.20.205.0 clusters


 [ 
https://issues.apache.org/jira/browse/HBASE-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4656.
--

   Resolution: Fixed
Fix Version/s: 0.92.0
 Assignee: stack
 Hadoop Flags: Reviewed

 Note how dfs.support.append has to be enabled in 0.20.205.0 clusters
 

 Key: HBASE-4656
 URL: https://issues.apache.org/jira/browse/HBASE-4656
 Project: HBase
  Issue Type: Task
Reporter: stack
Assignee: stack
 Fix For: 0.92.0

 Attachments: 4656.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4388) Second start after migration from 90 to trunk crashes


[ 
https://issues.apache.org/jira/browse/HBASE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134610#comment-13134610
 ] 

stack commented on HBASE-4388:
--

All unit tests passed for me.

 Second start after migration from 90 to trunk crashes
 -

 Key: HBASE-4388
 URL: https://issues.apache.org/jira/browse/HBASE-4388
 Project: HBase
  Issue Type: Bug
  Components: master, migration
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: stack
Priority: Blocker
 Fix For: 0.92.0

 Attachments: 4388-v2.txt, 4388-v3.txt, 4388-v4.txt, 4388.txt, 
 hbase-4388-root.dir.tgz, hbase-master-nase.log, meta.tgz


 I started a trunk cluster to upgrade from 90, inserted a ton of data, then 
 did a clean shutdown. When I started again, I got the following exception:
 11/09/13 12:29:09 INFO master.HMaster: Meta has HRI with HTDs. Updating meta 
 now.
 11/09/13 12:29:09 FATAL master.HMaster: Unhandled exception. Starting 
 shutdown.
 java.lang.NegativeArraySizeException: -102
 at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:147)
 at 
 org.apache.hadoop.hbase.HTableDescriptor.readFields(HTableDescriptor.java:606)
 at 
 org.apache.hadoop.hbase.migration.HRegionInfo090x.readFields(HRegionInfo090x.java:641)
 at 
 org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:133)
 at 
 org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:103)
 at 
 org.apache.hadoop.hbase.util.Writables.getHRegionInfoForMigration(Writables.java:228)
 at 
 org.apache.hadoop.hbase.catalog.MetaEditor.getHRegionInfoForMigration(MetaEditor.java:350)
 at 
 org.apache.hadoop.hbase.catalog.MetaEditor$1.visit(MetaEditor.java:273)
 at 
 org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:633)
 at 
 org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:255)
 at 
 org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:235)
 at 
 org.apache.hadoop.hbase.catalog.MetaEditor.updateMetaWithNewRegionInfo(MetaEditor.java:284)
 at 
 org.apache.hadoop.hbase.catalog.MetaEditor.migrateRootAndMeta(MetaEditor.java:298)
 at 
 org.apache.hadoop.hbase.master.HMaster.updateMetaWithNewHRI(HMaster.java:529)
 at 
 org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:472)
 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4578) NPE when altering a table that has moving regions


 [ 
https://issues.apache.org/jira/browse/HBASE-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4578.
--

  Resolution: Fixed
Hadoop Flags: Reviewed

Applied to trunk and branch.  Thanks for the patch gaojinchao.

 NPE when altering a table that has moving regions
 -

 Key: HBASE-4578
 URL: https://issues.apache.org/jira/browse/HBASE-4578
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: gaojinchao
Priority: Blocker
 Fix For: 0.92.0

 Attachments: HBASE-4578_Trunk_V1.patch, HBASE-4578_trial_Trunk.patch


 I'm still not a 100% sure on the source of this error, but here's what I was 
 able to get twice while altering a table that was doing a bunch of splits:
 {quote}
 2011-10-11 23:48:59,344 INFO 
 org.apache.hadoop.hbase.master.handler.SplitRegionHandler: Handled SPLIT 
 report); 
 parent=TestTable,0002608338,1318376880454.a75d6815fdfc513fb1c8aabe086c6763. 
 daughter 
 a=TestTable,0002608338,1318376938764.ef170ff6cd8695dc8aec92e542dc9ac1.daughter
  b=TestTable,0003301408,1318376938764.36eb2530341bd46888ede312c5559b5d.
 2011-10-11 23:49:09,579 DEBUG 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Ignoring table not 
 disabled exception for supporting online schema changes.
 2011-10-11 23:49:09,580 INFO 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Handling table 
 operation C_M_MODIFY_TABLE on table TestTable
 2011-10-11 23:49:09,612 INFO org.apache.hadoop.hbase.util.FSUtils: 
 TableInfoPath = hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo tmpPath = 
 hdfs://sv4r11s38:9100/hbase/TestTable/.tmp/.tableinfo.1318376949612
 2011-10-11 23:49:09,692 INFO org.apache.hadoop.hbase.util.FSUtils: 
 TableDescriptor stored. TableInfoPath = 
 hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo
 2011-10-11 23:49:09,693 INFO org.apache.hadoop.hbase.util.FSUtils: Updated 
 tableinfo=hdfs://sv4r11s38:9100/hbase/TestTable/.tableinfo to blah
 2011-10-11 23:49:09,695 INFO 
 org.apache.hadoop.hbase.master.handler.TableEventHandler: Bucketing regions 
 by region server...
 2011-10-11 23:49:09,695 DEBUG org.apache.hadoop.hbase.client.MetaScanner: 
 Scanning .META. starting at row=TestTable,,00 for max=2147483647 
 rows
 2011-10-11 23:49:09,709 DEBUG 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
 The connection to hconnection-0x132f043bbde02e9 has been closed.
 2011-10-11 23:49:09,709 ERROR org.apache.hadoop.hbase.executor.EventHandler: 
 Caught throwable while processing event C_M_MODIFY_TABLE
 java.lang.NullPointerException
   at java.util.TreeMap.getEntry(TreeMap.java:324)
   at java.util.TreeMap.containsKey(TreeMap.java:209)
   at 
 org.apache.hadoop.hbase.master.handler.TableEventHandler.reOpenAllRegions(TableEventHandler.java:114)
   at 
 org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:90)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:168)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
 {quote}
 The first time the shell reported that all the regions were updated 
 correctly, the second time it got stuck for a while:
 {quote}
 6/14 regions updated.
 0/14 regions updated.
 ...
 0/14 regions updated.
 2/16 regions updated.
 ...
 2/16 regions updated.
 8/9 regions updated.
 ...
 8/9 regions updated.
 {quote}
 After which I killed it, redid the alter and it worked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-2312) Possible data loss when RS goes into GC pause while rolling HLog


 [ 
https://issues.apache.org/jira/browse/HBASE-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-2312:
-

Fix Version/s: (was: 0.92.0)
   0.94.0

We don't have the hdfs support for this (0.20.205.0 or CDH).  Moving out to 
0.94.0.

 Possible data loss when RS goes into GC pause while rolling HLog
 

 Key: HBASE-2312
 URL: https://issues.apache.org/jira/browse/HBASE-2312
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 0.90.0
Reporter: Karthik Ranganathan
Assignee: Nicolas Spiegelberg
Priority: Critical
 Fix For: 0.94.0


 There is a very corner case when bad things could happen(ie data loss):
 1)RS #1 is going to roll its HLog - not yet created the new one, old one 
 will get no more writes
 2)RS #1 enters GC Pause of Death
 3)Master lists HLog files of RS#1 that is has to split as RS#1 is dead, 
 starts splitting
 4)RS #1 wakes up, created the new HLog (previous one was rolled) and 
 appends an edit - which is lost
 The following seems like a possible solution:
 1)Master detects RS#1 is dead
 2)The master renames the /hbase/.logs/regionserver name  directory to 
 something else (say /hbase/.logs/regionserver name-dead)
 3)Add mkdir support (as opposed to mkdirs) to HDFS - so that a file 
 create fails if the directory doesn't exist. Dhruba tells me this is very 
 doable.
 4)RS#1 comes back up and is not able create the new hlog. It restarts 
 itself.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master

2011-10-24 Thread Andrew Purtell (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-4070:
--

   Resolution: Fixed
Fix Version/s: 0.94.0
   0.92.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to trunk and 0.92

 [Coprocessors] Improve region server metrics to report loaded coprocessors to 
 master
 

 Key: HBASE-4070
 URL: https://issues.apache.org/jira/browse/HBASE-4070
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.3
Reporter: Mingjie Lai
Assignee: Eugene Koontz
 Fix For: 0.92.0, 0.94.0

 Attachments: HBASE-4070.patch, HBASE-4070.patch, HBASE-4070.patch, 
 HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg


 HBASE-3512 is about listing loaded cp classes at shell. To make it more 
 generic, we need a way to report this piece of information from region to 
 master (or just at region server level). So later on, we can display the 
 loaded class names at shell as well as web console. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4374) Up default regions size from 256M to 1G

2011-10-24 Thread Jean-Daniel Cryans (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134622#comment-13134622
 ] 

Jean-Daniel Cryans commented on HBASE-4374:
---

+1

 Up default regions size from 256M to 1G
 ---

 Key: HBASE-4374
 URL: https://issues.apache.org/jira/browse/HBASE-4374
 Project: HBase
  Issue Type: Task
Reporter: stack
Priority: Blocker
 Fix For: 0.92.0


 HBASE-4365 has some discussion of why we default for a table should tend to 
 fewer bigger regions.  It doesn't look like this issue will be done for 0.92. 
  For 0.92, lets up default region size from 256M to 1G and talk up pre-split 
 on table creation in manual.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4648) Bytes.toBigDecimal() doesn't use offset

2011-10-24 Thread Lars Hofhansl (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134625#comment-13134625
 ] 

Lars Hofhansl commented on HBASE-4648:
--

This:
{code}
  public static BigDecimal toBigDecimal(byte[] bytes, int offset) {
return toBigDecimal(bytes, offset, bytes.length);
  }
{code}

Is also wrong. Should be:
{code}
return toBigDecimal(bytes, offset, bytes.length-offset);
{code}


 Bytes.toBigDecimal() doesn't use offset
 ---

 Key: HBASE-4648
 URL: https://issues.apache.org/jira/browse/HBASE-4648
 Project: HBase
  Issue Type: Bug
  Components: util
Affects Versions: 0.90.4
 Environment: Java 1.6.0_26, Mac OS X 10.7 and CentOS 6
Reporter: Bryan Keller

 The Bytes.toBigDecimal(byte[], offset, len) method does not use the offset, 
 thus you will get an incorrect result for the BigDecimal unless the 
 BigDecimal's bytes are at the beginning of the byte array.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4254) Get tests passing on Hadoop 23

2011-10-24 Thread Nicolas Spiegelberg (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4254:
-

Priority: Major  (was: Critical)

Knocking this down to major from critical.

 Get tests passing on Hadoop 23
 --

 Key: HBASE-4254
 URL: https://issues.apache.org/jira/browse/HBASE-4254
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: 0.92.0


 Currently some 30 or so tests are failing on the HBase-trunk-on-hadoop-23 
 build. It looks like most are reflection-based issues.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3512) Coprocessors: Shell support for listing currently loaded coprocessor set

2011-10-24 Thread Andrew Purtell (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-3512:
--

   Resolution: Fixed
Fix Version/s: 0.94.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to trunk and 0.92 branch.

 Coprocessors: Shell support for listing currently loaded coprocessor set
 

 Key: HBASE-3512
 URL: https://issues.apache.org/jira/browse/HBASE-3512
 Project: HBase
  Issue Type: Improvement
  Components: coprocessors
Reporter: Andrew Purtell
Assignee: Eugene Koontz
 Fix For: 0.92.0, 0.94.0

 Attachments: HBASE-3512-only.patch, HBASE-3512-only.patch, 
 HBASE-3512.patch, HBASE-3512.patch, HBASE-3512.patch, hbase-shell-session.txt


 Add support to the shell for listing the coprocessors loaded globally on the 
 regionserver and those loaded on a per-table basis.
 Perhaps by extending the 'status' command.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2312) Possible data loss when RS goes into GC pause while rolling HLog


[ 
https://issues.apache.org/jira/browse/HBASE-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134629#comment-13134629
 ] 

Nicolas Spiegelberg commented on HBASE-2312:


@stack: since we have proper reflection, can we not apply this jira  mark the 
test as broken?  that way, customers can have this fix if they upgrade to the 
proper version of HDFS instead of requiring an HBase upgrade as well

 Possible data loss when RS goes into GC pause while rolling HLog
 

 Key: HBASE-2312
 URL: https://issues.apache.org/jira/browse/HBASE-2312
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 0.90.0
Reporter: Karthik Ranganathan
Assignee: Nicolas Spiegelberg
Priority: Critical
 Fix For: 0.94.0


 There is a very corner case when bad things could happen(ie data loss):
 1)RS #1 is going to roll its HLog - not yet created the new one, old one 
 will get no more writes
 2)RS #1 enters GC Pause of Death
 3)Master lists HLog files of RS#1 that is has to split as RS#1 is dead, 
 starts splitting
 4)RS #1 wakes up, created the new HLog (previous one was rolled) and 
 appends an edit - which is lost
 The following seems like a possible solution:
 1)Master detects RS#1 is dead
 2)The master renames the /hbase/.logs/regionserver name  directory to 
 something else (say /hbase/.logs/regionserver name-dead)
 3)Add mkdir support (as opposed to mkdirs) to HDFS - so that a file 
 create fails if the directory doesn't exist. Dhruba tells me this is very 
 doable.
 4)RS#1 comes back up and is not able create the new hlog. It restarts 
 itself.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4648) Bytes.toBigDecimal() doesn't use offset

2011-10-24 Thread Bryan Keller (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134628#comment-13134628
 ] 

Bryan Keller commented on HBASE-4648:
-

I was actually thinking that method isn't useful, it seems it should be 
removed, as the BigDecimal serialization is variable-length. It would only be 
used if the BigDecimal comes at the end of the byte array.

 Bytes.toBigDecimal() doesn't use offset
 ---

 Key: HBASE-4648
 URL: https://issues.apache.org/jira/browse/HBASE-4648
 Project: HBase
  Issue Type: Bug
  Components: util
Affects Versions: 0.90.4
 Environment: Java 1.6.0_26, Mac OS X 10.7 and CentOS 6
Reporter: Bryan Keller

 The Bytes.toBigDecimal(byte[], offset, len) method does not use the offset, 
 thus you will get an incorrect result for the BigDecimal unless the 
 BigDecimal's bytes are at the beginning of the byte array.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4648) Bytes.toBigDecimal() doesn't use offset

2011-10-24 Thread Bryan Keller (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Keller updated HBASE-4648:


Attachment: bigdec.patch

Here is the patch. I also added some tests for testing serialization with 
offsets for other data types.

 Bytes.toBigDecimal() doesn't use offset
 ---

 Key: HBASE-4648
 URL: https://issues.apache.org/jira/browse/HBASE-4648
 Project: HBase
  Issue Type: Bug
  Components: util
Affects Versions: 0.90.4
 Environment: Java 1.6.0_26, Mac OS X 10.7 and CentOS 6
Reporter: Bryan Keller
 Attachments: bigdec.patch


 The Bytes.toBigDecimal(byte[], offset, len) method does not use the offset, 
 thus you will get an incorrect result for the BigDecimal unless the 
 BigDecimal's bytes are at the beginning of the byte array.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4648) Bytes.toBigDecimal() doesn't use offset

2011-10-24 Thread Bryan Keller (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Keller updated HBASE-4648:


Attachment: bigdec2.patch

Sorry bad patch, here's a new one.

 Bytes.toBigDecimal() doesn't use offset
 ---

 Key: HBASE-4648
 URL: https://issues.apache.org/jira/browse/HBASE-4648
 Project: HBase
  Issue Type: Bug
  Components: util
Affects Versions: 0.90.4
 Environment: Java 1.6.0_26, Mac OS X 10.7 and CentOS 6
Reporter: Bryan Keller
 Attachments: bigdec.patch, bigdec2.patch


 The Bytes.toBigDecimal(byte[], offset, len) method does not use the offset, 
 thus you will get an incorrect result for the BigDecimal unless the 
 BigDecimal's bytes are at the beginning of the byte array.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-24 Thread Amitanand Aiyer (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amitanand Aiyer updated HBASE-4645:
---

Attachment: 4645-v9.diff

v9 of the revision from https://reviews.apache.org/r/2524/diff

 Edits Log recovery losing data across column families
 -

 Key: HBASE-4645
 URL: https://issues.apache.org/jira/browse/HBASE-4645
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer
 Attachments: 4645-v9.diff


 There is a data loss happening (for some of the column families) when we do 
 the replay logs.
 The bug seems to be from the fact that during replay-logs we only choose to 
 replay
 the logs from the maximumSequenceID across *ALL* the stores. This is wrong. 
 If a
 column family is ahead of others (because the crash happened before all the 
 column
 families were flushed), then we lose data for the column families that have 
 not yet
 caught up.
 The correct logic for replay should begin the replay from the minimum across 
 the
 maximum in each store. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4645) Edits Log recovery losing data across column families

2011-10-24 Thread Amitanand Aiyer (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amitanand Aiyer updated HBASE-4645:
---

Status: Patch Available  (was: Open)

 Edits Log recovery losing data across column families
 -

 Key: HBASE-4645
 URL: https://issues.apache.org/jira/browse/HBASE-4645
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.89.20100924, 0.92.0
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer
 Attachments: 4645-v9.diff


 There is a data loss happening (for some of the column families) when we do 
 the replay logs.
 The bug seems to be from the fact that during replay-logs we only choose to 
 replay
 the logs from the maximumSequenceID across *ALL* the stores. This is wrong. 
 If a
 column family is ahead of others (because the crash happened before all the 
 column
 families were flushed), then we lose data for the column families that have 
 not yet
 caught up.
 The correct logic for replay should begin the replay from the minimum across 
 the
 maximum in each store. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4657) Improve the efficiency of our MR jobs with a few configurations

2011-10-24 Thread Jean-Daniel Cryans (Created) (JIRA)

Improve the efficiency of our MR jobs with a few configurations
---

 Key: HBASE-4657
 URL: https://issues.apache.org/jira/browse/HBASE-4657
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
 Fix For: 0.92.0


This is a low hanging fruit, some of our MR jobs like RowCounter and CopyTable 
don't even setCacheBlocks on the scan object which out of the box completely 
screws up a running system. Another thing would be to disable speculative 
execution.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4591) TTL for old HLogs should be calculated from last modification time.


[ 
https://issues.apache.org/jira/browse/HBASE-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134653#comment-13134653
 ] 

Hudson commented on HBASE-4591:
---

Integrated in HBase-TRUNK #2365 (See 
[https://builds.apache.org/job/HBase-TRUNK/2365/])
HBASE-4591 (2) Fix broken unit test
HBASE-4591 TTL for old HLogs should be calculated from last modification time.

nspiegelberg : 
Files : 
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java

nspiegelberg : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java


 TTL for old HLogs should be calculated from last modification time.
 ---

 Key: HBASE-4591
 URL: https://issues.apache.org/jira/browse/HBASE-4591
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.89.20100621
Reporter: Madhuwanti Vaidya
Assignee: Madhuwanti Vaidya
Priority: Minor
 Fix For: 0.92.0, 0.94.0

 Attachments: 
 0007-HBASE-4591-Increase-TTL-for-old-HLogs-and-change-TTL.patch, 
 HBASE-4591-2.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4437) Update hadoop in 0.92 (0.20.205?)


[ 
https://issues.apache.org/jira/browse/HBASE-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134654#comment-13134654
 ] 

Hudson commented on HBASE-4437:
---

Integrated in HBase-TRUNK #2365 (See 
[https://builds.apache.org/job/HBase-TRUNK/2365/])
HBASE-4437 Update hadoop in 0.92 (0.20.205?)

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/pom.xml
* 
/hbase/trunk/src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon
* 
/hbase/trunk/src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon
* /hbase/trunk/src/main/resources/hbase-webapps/master/table.jsp
* /hbase/trunk/src/main/resources/hbase-webapps/master/tablesDetailed.jsp
* /hbase/trunk/src/main/resources/hbase-webapps/master/zk.jsp


 Update hadoop in 0.92 (0.20.205?)
 -

 Key: HBASE-4437
 URL: https://issues.apache.org/jira/browse/HBASE-4437
 Project: HBase
  Issue Type: Task
Reporter: stack
Assignee: stack
 Fix For: 0.92.0

 Attachments: 4437.txt


 We ship with branch-0.20-append a few versions back from the tip.  If 205 
 comes out and hbase works on it, we should ship 0.92 with it (while also 
 ensuring it work w/ 0.22 and 0.23 branches).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4658) Put attributes are not exposed via the ThriftServer

2011-10-24 Thread dhruba borthakur (Created) (JIRA)

Put attributes are not exposed via the ThriftServer
---

 Key: HBASE-4658
 URL: https://issues.apache.org/jira/browse/HBASE-4658
 Project: HBase
  Issue Type: Bug
  Components: thrift
Reporter: dhruba borthakur
Assignee: dhruba borthakur


The Put api also takes in a bunch of arbitrary attributes that an application 
can use to associate metadata with each put operation. This is not exposed via 
Thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4374) Up default regions size from 256M to 1G


 [ 
https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4374:
-

Attachment: 4374.txt

Flush size of 128M and split size of 1G.

 Up default regions size from 256M to 1G
 ---

 Key: HBASE-4374
 URL: https://issues.apache.org/jira/browse/HBASE-4374
 Project: HBase
  Issue Type: Task
Reporter: stack
Priority: Blocker
 Fix For: 0.92.0

 Attachments: 4374.txt


 HBASE-4365 has some discussion of why we default for a table should tend to 
 fewer bigger regions.  It doesn't look like this issue will be done for 0.92. 
  For 0.92, lets up default region size from 256M to 1G and talk up pre-split 
 on table creation in manual.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4374) Up default regions size from 256M to 1G


 [ 
https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4374.
--

  Resolution: Fixed
Release Note: Upped default split size to 1G from 256M and the flush size 
from 64M to 128M.

 Up default regions size from 256M to 1G
 ---

 Key: HBASE-4374
 URL: https://issues.apache.org/jira/browse/HBASE-4374
 Project: HBase
  Issue Type: Task
Reporter: stack
Priority: Blocker
 Fix For: 0.92.0

 Attachments: 4374.txt


 HBASE-4365 has some discussion of why we default for a table should tend to 
 fewer bigger regions.  It doesn't look like this issue will be done for 0.92. 
  For 0.92, lets up default region size from 256M to 1G and talk up pre-split 
 on table creation in manual.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4374) Up default regions size from 256M to 1G


[ 
https://issues.apache.org/jira/browse/HBASE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134729#comment-13134729
 ] 

stack commented on HBASE-4374:
--

Committed to branch and trunk.

 Up default regions size from 256M to 1G
 ---

 Key: HBASE-4374
 URL: https://issues.apache.org/jira/browse/HBASE-4374
 Project: HBase
  Issue Type: Task
Reporter: stack
Priority: Blocker
 Fix For: 0.92.0

 Attachments: 4374.txt


 HBASE-4365 has some discussion of why we default for a table should tend to 
 fewer bigger regions.  It doesn't look like this issue will be done for 0.92. 
  For 0.92, lets up default region size from 256M to 1G and talk up pre-split 
 on table creation in manual.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4659) SingleColumnValueFilter is not working correctly

2011-10-24 Thread guozhw (Created) (JIRA)

SingleColumnValueFilter is not working correctly


 Key: HBASE-4659
 URL: https://issues.apache.org/jira/browse/HBASE-4659
 Project: HBase
  Issue Type: Bug
  Components: filters
Affects Versions: 0.90.4
Reporter: guozhw


scan.addColumn(Bytes.toBytes(Info), Bytes.toBytes(value));
scan.setMaxVersions();
SingleColumnValueFilter scvf = new 
SingleColumnValueFilter(Bytes.toBytes(Info), Bytes.toBytes(value), 
CompareOp.EQUAL, Bytes.toBytes(test));
scvf.setLatestVersionOnly(false);
scan.setFilter(scvf);


it can not filter previous versions

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4660) Place to publish RegionServer information such as webuiport and coprocessors loaded

2011-10-24 Thread stack (Created) (JIRA)

Place to publish RegionServer information such as webuiport and coprocessors 
loaded
---

 Key: HBASE-4660
 URL: https://issues.apache.org/jira/browse/HBASE-4660
 Project: HBase
  Issue Type: Bug
Reporter: stack


HBASE-4070 added loaded CoProcessors to HServerLoad which seems like wrong 
place to carry this info.  We need a locus for static info of this type such as 
loaded CoProcessors and webuiport as well as stuff like how many cpus, RAM, 
etc: e.g. in regionserver znode or available on invocation of an HRegionServer 
method (master can ask HRegionServer when it needs it).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master


[ 
https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134734#comment-13134734
 ] 

stack commented on HBASE-4070:
--

Here was my last comment on the patch over on RB in response to Eugene's Hi 
Michael, As Ted mentioned, HServerInfo is deprecated (per 
https://issues.apache.org/jira/browse/HBASE-1502):

{code}
Michael Stack 5 days, 4 hours ago (October 19th, 2011, 11:34 p.m.)  
Yes but that is not reason to shoe horn it in elsewhere (smile).

There is another server attribute that is currently without a home (webuiport) 
and I think there are other a bunch of server attrbutes -- as opposed to server 
load -- that servers might export and have aggregated over in the master and/or 
used calculating cluster balance and the like.

We could let this patch go in as is or we could build the server attributes 
mechanism -- ServerInfoV2 or json in regionserver znode, etc. -- and put stuff 
like this in there.
{code}

I think it a little perverse passing loaded coprocessors as part of 
HServerLoad.  I opened HBASE-4660 to address it.

 [Coprocessors] Improve region server metrics to report loaded coprocessors to 
 master
 

 Key: HBASE-4070
 URL: https://issues.apache.org/jira/browse/HBASE-4070
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.90.3
Reporter: Mingjie Lai
Assignee: Eugene Koontz
 Fix For: 0.92.0, 0.94.0

 Attachments: HBASE-4070.patch, HBASE-4070.patch, HBASE-4070.patch, 
 HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg


 HBASE-3512 is about listing loaded cp classes at shell. To make it more 
 generic, we need a way to report this piece of information from region to 
 master (or just at region server level). So later on, we can display the 
 loaded class names at shell as well as web console. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-2312) Possible data loss when RS goes into GC pause while rolling HLog