[jira] [Updated] (HBASE-3134) [replication] Add the ability to enable/disable streams
[ https://issues.apache.org/jira/browse/HBASE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teruyoshi Zenmyo updated HBASE-3134: Attachment: HBASE-3134.patch The patch is updated. A test and some comments and typos are modified. [replication] Add the ability to enable/disable streams --- Key: HBASE-3134 URL: https://issues.apache.org/jira/browse/HBASE-3134 Project: HBase Issue Type: New Feature Components: replication Reporter: Jean-Daniel Cryans Assignee: Teruyoshi Zenmyo Priority: Minor Labels: replication Fix For: 0.94.0 Attachments: HBASE-3134.patch, HBASE-3134.patch, HBASE-3134.patch This jira was initially in the scope of HBASE-2201, but was pushed out since it has low value compared to the required effort (and when want to ship 0.90.0 rather soonish). We need to design a way to enable/disable replication streams in a determinate fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5330) TestCompactSelection - adding 2 test cases to testCompactionRatio
[ https://issues.apache.org/jira/browse/HBASE-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200459#comment-13200459 ] Doug Meil commented on HBASE-5330: -- Thanks Nicholas. I'll give #1 and #2 a shot. Especially for #3, the reason I was adding to the tests in the first place was to describe what is happening as opposed to what should be happening. Basically, I want to get a better description of compaction in the book, so I figured that the unit tests were the best place to start. TestCompactSelection - adding 2 test cases to testCompactionRatio - Key: HBASE-5330 URL: https://issues.apache.org/jira/browse/HBASE-5330 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: TestCompactSelection_hbase_5330.java.patch There were three existing assertions in TestCompactSelection testCompactionRatio that did max # of files assertions... {code} assertEquals(maxFiles, store.compactSelection(sfCreate(7,6,5,4,3,2,1)).getFilesToCompact().size()); {code} ... and for references ... {code} assertEquals(maxFiles, store.compactSelection(sfCreate(true, 7,6,5,4,3,2,1)).getFilesToCompact().size()); {code} ... but they didn't assert against which StoreFiles got selected. While the number of StoreFiles is the same, the files selected are actually different, and I thought that there should be explicit assertions showing that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5329) addRowLock() may allocate duplicate lock id, causing the client to be blocked
[ https://issues.apache.org/jira/browse/HBASE-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200479#comment-13200479 ] Lars Hofhansl commented on HBASE-5329: -- Even if you get 1000 scanners per second you'd have to do that for 1 trillion (1.000.000.000) years to come into the ballpark of 2^64. The laws of big numbers are not intuitive. It is billions times more likely that your power fails, there're random bit errors, your harddrive dies, or datacenter is hit my a meteorite :). From that viewpoint this is in fact not possible, because your hardware is not going to live long enough to experience this. I really think we do not have to change this. addRowLock() may allocate duplicate lock id, causing the client to be blocked - Key: HBASE-5329 URL: https://issues.apache.org/jira/browse/HBASE-5329 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.3 Environment: Red Hat Enterprise Linux Server release 5.4 Reporter: liaoxiangui Assignee: Zhihong Yu Priority: Critical {code} protected long addRowLock(Integer r, HRegion region) throws LeaseStillHeldException { long lockId = -1L; lockId = rand.nextLong(); //!!!may generate duplicated id,bug? String lockName = String.valueOf(lockId); rowlocks.put(lockName, r); this.leases.createLease(lockName, new RowLockListener(lockName, region)); return lockId; } {code} In addRowLock(),rand may generate duplicated lock id, it may cause regionserver throw exception(Leases$LeaseStillHeldException).The client will be blocked until old rowlock is released. {code} 2012-02-03 15:21:50,084 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Error obtaining row lock (fsOk: true) org.apache.hadoop.hbase.regionserver.Leases$LeaseStillHeldException at org.apache.hadoop.hbase.regionserver.Leases.createLease(Leases.java:150) at org.apache.hadoop.hbase.regionserver.HRegionServer.addRowLock(HRegionServer.java:1986) at org.apache.hadoop.hbase.regionserver.HRegionServer.lockRow(HRegionServer.java:1963) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5317) Fix TestHFileOutputFormat to work against hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200575#comment-13200575 ] Jimmy Xiang commented on HBASE-5317: @Ted, did you run it with Hadoop 0.23? Fix TestHFileOutputFormat to work against hadoop 0.23 - Key: HBASE-5317 URL: https://issues.apache.org/jira/browse/HBASE-5317 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0, 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Attachments: HBASE-5317-v0.patch Running mvn -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat yields this on 0.92: Failed tests: testColumnFamilyCompression(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): HFile for column family info-A not found Tests in error: test_TIMERANGE(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): /home/gchanan/workspace/apache92/target/test-data/276cbd0c-c771-4f81-9ba8-c464c9dd7486/test_TIMERANGE_present/_temporary/0/_temporary/_attempt_200707121733_0001_m_00_0 (Is a directory) testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable It looks like on trunk, this also results in an error: testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable I have a patch that fixes testColumnFamilyCompression and test_TIMERANGE, but haven't fixed the other 3 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5330) TestCompactSelection - adding 2 test cases to testCompactionRatio
[ https://issues.apache.org/jira/browse/HBASE-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200578#comment-13200578 ] Doug Meil commented on HBASE-5330: -- Hey Nicholas, I initially mis-read this comment We need to change compactSelection to return whether the operation is a major compaction or not and now I see what you're talking about: there is no isMajorCompaction in CompactSelection. TestCompactSelection - adding 2 test cases to testCompactionRatio - Key: HBASE-5330 URL: https://issues.apache.org/jira/browse/HBASE-5330 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: TestCompactSelection_hbase_5330.java.patch There were three existing assertions in TestCompactSelection testCompactionRatio that did max # of files assertions... {code} assertEquals(maxFiles, store.compactSelection(sfCreate(7,6,5,4,3,2,1)).getFilesToCompact().size()); {code} ... and for references ... {code} assertEquals(maxFiles, store.compactSelection(sfCreate(true, 7,6,5,4,3,2,1)).getFilesToCompact().size()); {code} ... but they didn't assert against which StoreFiles got selected. While the number of StoreFiles is the same, the files selected are actually different, and I thought that there should be explicit assertions showing that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5317) Fix TestHFileOutputFormat to work against hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200580#comment-13200580 ] Zhihong Yu commented on HBASE-5317: --- Here was the command I used: {code} mvn clean -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat {code} Fix TestHFileOutputFormat to work against hadoop 0.23 - Key: HBASE-5317 URL: https://issues.apache.org/jira/browse/HBASE-5317 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0, 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Attachments: HBASE-5317-v0.patch Running mvn -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat yields this on 0.92: Failed tests: testColumnFamilyCompression(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): HFile for column family info-A not found Tests in error: test_TIMERANGE(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): /home/gchanan/workspace/apache92/target/test-data/276cbd0c-c771-4f81-9ba8-c464c9dd7486/test_TIMERANGE_present/_temporary/0/_temporary/_attempt_200707121733_0001_m_00_0 (Is a directory) testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable It looks like on trunk, this also results in an error: testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable I have a patch that fixes testColumnFamilyCompression and test_TIMERANGE, but haven't fixed the other 3 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5322) CLONE - getScanner hangs with some startRows that are found if scanning entire table
[ https://issues.apache.org/jira/browse/HBASE-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200609#comment-13200609 ] Lars Hofhansl commented on HBASE-5322: -- No exceptions on client or server? What do you mean with used the same client scanner? Lastly, does this happen every time? Any extra info you can provide might be helpful. CLONE - getScanner hangs with some startRows that are found if scanning entire table Key: HBASE-5322 URL: https://issues.apache.org/jira/browse/HBASE-5322 Project: HBase Issue Type: Bug Affects Versions: 0.90.4 Reporter: Karthik Pandian Priority: Blocker I have a hbase table which holds data for more than 10GB. Now I used the same client scanner to scan which fails and reports, Could not seek StoreFileScanner[HFileScanner for reader reader=hdfs. This issue occurs only for the table which holds huge data and not for tables holding small data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5320) Create client API to handle HBase maintenance gracefully
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200611#comment-13200611 ] Lars Hofhansl commented on HBASE-5320: -- I have no concrete plans, yet. There are simple ideas of # just letting the client wait it out (blocking the client) or # just check whether ZK/Master is up, let the app react to that I am not following why there needs to be another server to determine whether a service down. Or are you planning to preannounce outages through this extra server? Or to distinguish between an outage and a planned maintenance? Create client API to handle HBase maintenance gracefully Key: HBASE-5320 URL: https://issues.apache.org/jira/browse/HBASE-5320 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3134) [replication] Add the ability to enable/disable streams
[ https://issues.apache.org/jira/browse/HBASE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200615#comment-13200615 ] Lars Hofhansl commented on HBASE-3134: -- I think this is good to go. Let's get J-D to look at it once. [replication] Add the ability to enable/disable streams --- Key: HBASE-3134 URL: https://issues.apache.org/jira/browse/HBASE-3134 Project: HBase Issue Type: New Feature Components: replication Reporter: Jean-Daniel Cryans Assignee: Teruyoshi Zenmyo Priority: Minor Labels: replication Fix For: 0.94.0 Attachments: HBASE-3134.patch, HBASE-3134.patch, HBASE-3134.patch This jira was initially in the scope of HBASE-2201, but was pushed out since it has low value compared to the required effort (and when want to ship 0.90.0 rather soonish). We need to design a way to enable/disable replication streams in a determinate fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5229) Explore building blocks for multi-row local transactions.
[ https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200616#comment-13200616 ] Lars Hofhansl commented on HBASE-5229: -- Is there still general trepidation around this? The patch is very lightweight and does not change any existing functionality. With HBASE-5304 it can be quite helpful to provide local transaction to help in multi-tenant settings. We can possibly call out the advancedness of this feature, maybe by having an advanced htable (or something like this). Not sure that is worth splitting the API, though. Explore building blocks for multi-row local transactions. --- Key: HBASE-5229 URL: https://issues.apache.org/jira/browse/HBASE-5229 Project: HBase Issue Type: New Feature Components: client, regionserver Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 5229-multiRow-v2.txt, 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt, 5229.txt HBase should provide basic building blocks for multi-row local transactions. Local means that we do this by co-locating the data. Global (cross region) transactions are not discussed here. After a bit of discussion two solutions have emerged: 1. Keep the row-key for determining grouping and location and allow efficient intra-row scanning. A client application would then model tables as HBase-rows. 2. Define a prefix-length in HTableDescriptor that defines a grouping of rows. Regions will then never be split inside a grouping prefix. #1 is true to the current storage paradigm of HBase. #2 is true to the current client side API. I will explore these two with sample patches here. Was: As discussed (at length) on the dev mailing list with the HBASE-3584 and HBASE-5203 committed, supporting atomic cross row transactions within a region becomes simple. I am aware of the hesitation about the usefulness of this feature, but we have to start somewhere. Let's use this jira for discussion, I'll attach a patch (with tests) momentarily to make this concrete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5317) Fix TestHFileOutputFormat to work against hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200632#comment-13200632 ] Gregory Chanan commented on HBASE-5317: --- @Ted: your output is correct. There is another patch required to get past the TestTable error and to the Error starting MRAppMaster. I didn't include it because it doesn't solve the issue, only lets you further along. I wanted to leave the patch in a fresh state that actually solves the issue addressed by the JIRA. Perhaps the best thing to do is narrow the scope of this JIRA (to just get test_TIMERANGE and testColumnFamilyCompression to pass) and then open a new JIRA to solve the remaining issues. Do you think that makes the most sense? Fix TestHFileOutputFormat to work against hadoop 0.23 - Key: HBASE-5317 URL: https://issues.apache.org/jira/browse/HBASE-5317 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0, 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Attachments: HBASE-5317-v0.patch Running mvn -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat yields this on 0.92: Failed tests: testColumnFamilyCompression(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): HFile for column family info-A not found Tests in error: test_TIMERANGE(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): /home/gchanan/workspace/apache92/target/test-data/276cbd0c-c771-4f81-9ba8-c464c9dd7486/test_TIMERANGE_present/_temporary/0/_temporary/_attempt_200707121733_0001_m_00_0 (Is a directory) testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable It looks like on trunk, this also results in an error: testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable I have a patch that fixes testColumnFamilyCompression and test_TIMERANGE, but haven't fixed the other 3 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5317) Fix TestHFileOutputFormat to work against hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5317: -- Status: Patch Available (was: Open) Fix TestHFileOutputFormat to work against hadoop 0.23 - Key: HBASE-5317 URL: https://issues.apache.org/jira/browse/HBASE-5317 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.92.0, 0.94.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Attachments: HBASE-5317-v0.patch Running mvn -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat yields this on 0.92: Failed tests: testColumnFamilyCompression(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): HFile for column family info-A not found Tests in error: test_TIMERANGE(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): /home/gchanan/workspace/apache92/target/test-data/276cbd0c-c771-4f81-9ba8-c464c9dd7486/test_TIMERANGE_present/_temporary/0/_temporary/_attempt_200707121733_0001_m_00_0 (Is a directory) testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable It looks like on trunk, this also results in an error: testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable I have a patch that fixes testColumnFamilyCompression and test_TIMERANGE, but haven't fixed the other 3 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5212) Fix test TestTableMapReduce against 0.23.
[ https://issues.apache.org/jira/browse/HBASE-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200636#comment-13200636 ] Hudson commented on HBASE-5212: --- Integrated in HBase-0.92-security #90 (See [https://builds.apache.org/job/HBase-0.92-security/90/]) HBASE-5212 Fix test TestTableMapReduce against 0.23 (Ted and Gregory) jmhsieh : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/pom.xml * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java Fix test TestTableMapReduce against 0.23. - Key: HBASE-5212 URL: https://issues.apache.org/jira/browse/HBASE-5212 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Mahadev konar Assignee: Gregory Chanan Fix For: 0.94.0, 0.92.1 Attachments: 5212-v2.txt, HBASE-5212-v3.patch, HBASE-5212.patch As reported by Andrew on the hadoop mailing list, mvn -Dhadoop.profile=23 clean test -Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce fails on 0.92 branch. There are minor changes to HBase poms required to fix that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5282) Possible file handle leak with truncated HLog file.
[ https://issues.apache.org/jira/browse/HBASE-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200635#comment-13200635 ] Hudson commented on HBASE-5282: --- Integrated in HBase-0.92-security #90 (See [https://builds.apache.org/job/HBase-0.92-security/90/]) HBASE-5282 Possible file handle leak with truncated HLog file jmhsieh : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java Possible file handle leak with truncated HLog file. --- Key: HBASE-5282 URL: https://issues.apache.org/jira/browse/HBASE-5282 Project: HBase Issue Type: Bug Affects Versions: 0.94.0, 0.90.5, 0.92.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.94.0, 0.92.1 Attachments: hbase-5282.patch, hbase-5282.v2.patch When debugging hbck, found that the code responsible for this exception can leak open file handles. {code} 12/01/15 05:58:11 INFO regionserver.HRegion: Replaying edits from hdfs://haus01. sf.cloudera.com:56020/hbase-jon/test5/98a1e7255731aae44b3836641840113e/recovered .edits/3211315; minSequenceid=3214658 12/01/15 05:58:11 ERROR handler.OpenRegionHandler: Failed open of region=test5,8 \x90\x00\x00\x00\x00\x00\x00/05_0,1326597390073.98a1e7255731aae44b3836641840 113e. java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:250) at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:299) at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:320) at org.apache.hadoop.io.Text.readString(Text.java:400) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1486) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1437) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1424) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1419) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.init(SequenceFileLogReader.java:57) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:158) at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:572) at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:1940) at org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:1896) at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:366) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2661) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2647) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:312) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:99) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:158) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5271) Result.getValue and Result.getColumnLatest return the wrong column.
[ https://issues.apache.org/jira/browse/HBASE-5271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200638#comment-13200638 ] Hudson commented on HBASE-5271: --- Integrated in HBase-0.92-security #90 (See [https://builds.apache.org/job/HBase-0.92-security/90/]) HBASE-5271 Result.getValue and Result.getColumnLatest return the wrong column (Ghais Issa) tedyu : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/KeyValue.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java Result.getValue and Result.getColumnLatest return the wrong column. --- Key: HBASE-5271 URL: https://issues.apache.org/jira/browse/HBASE-5271 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.90.5 Reporter: Ghais Issa Assignee: Ghais Issa Fix For: 0.94.0, 0.90.7, 0.92.1 Attachments: 5271-90.txt, 5271-v2.txt, fixKeyValueMatchingColumn.diff, testGetValue.diff In the following example result.getValue returns the wrong column KeyValue kv = new KeyValue(Bytes.toBytes(r), Bytes.toBytes(24), Bytes.toBytes(2), Bytes.toBytes(7L)); Result result = new Result(new KeyValue[] { kv }); System.out.println(Bytes.toLong(result.getValue(Bytes.toBytes(2), Bytes.toBytes(2; //prints 7. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5307) Unable to gracefully decommission a node because of script error
[ https://issues.apache.org/jira/browse/HBASE-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200637#comment-13200637 ] Hudson commented on HBASE-5307: --- Integrated in HBase-0.92-security #90 (See [https://builds.apache.org/job/HBase-0.92-security/90/]) HBASE-5307 Unable to gracefully decommission a node because of script error stack : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/bin/region_mover.rb Unable to gracefully decommission a node because of script error Key: HBASE-5307 URL: https://issues.apache.org/jira/browse/HBASE-5307 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.92.0 Reporter: YiFeng Jiang Fix For: 0.92.1 Attachments: region_mover_HBASE-5307-0.92.patch Unable to gracefully decommission a node because NameError occurred in region_mover.rb {code} $ bin/graceful_stop.sh ip-10-160-226-84.us-west-1.compute.internal ... NameError: no constructorfor arguments (org.jruby.RubyString) on Java::OrgApacheHadoopHbase::HServerAddress available overloads: (org.apache.hadoop.hbase.HServerAddress) (java.net.InetSocketAddress) getRegions at /usr/local/hbase/current/bin/region_mover.rb:254 unloadRegions at /usr/local/hbase/current/bin/region_mover.rb:314 (root) at /usr/local/hbase/current/bin/region_mover.rb:430 Unloaded ip-10-160-226-84.us-west-1.compute.internal region(s) ip-10-160-226-84.us-west-1.compute.internal: stopping regionserver.. {code} The reason is the region_mover.rb calls wrong HBase APIs to try to establish a connection to the region server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5317) Fix TestHFileOutputFormat to work against hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200644#comment-13200644 ] Hadoop QA commented on HBASE-5317: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513229/HBASE-5317-v0.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -136 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 156 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/904//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/904//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/904//console This message is automatically generated. Fix TestHFileOutputFormat to work against hadoop 0.23 - Key: HBASE-5317 URL: https://issues.apache.org/jira/browse/HBASE-5317 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0, 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Attachments: HBASE-5317-v0.patch Running mvn -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat yields this on 0.92: Failed tests: testColumnFamilyCompression(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): HFile for column family info-A not found Tests in error: test_TIMERANGE(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): /home/gchanan/workspace/apache92/target/test-data/276cbd0c-c771-4f81-9ba8-c464c9dd7486/test_TIMERANGE_present/_temporary/0/_temporary/_attempt_200707121733_0001_m_00_0 (Is a directory) testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable It looks like on trunk, this also results in an error: testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable I have a patch that fixes testColumnFamilyCompression and test_TIMERANGE, but haven't fixed the other 3 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5317) Fix TestHFileOutputFormat to work against hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200650#comment-13200650 ] Zhihong Yu commented on HBASE-5317: --- The subject of this JIRA is for all of TestHFileOutputFormat to pass for 0.23 I think we should try to achieve that goal. Fix TestHFileOutputFormat to work against hadoop 0.23 - Key: HBASE-5317 URL: https://issues.apache.org/jira/browse/HBASE-5317 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0, 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Attachments: HBASE-5317-v0.patch Running mvn -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat yields this on 0.92: Failed tests: testColumnFamilyCompression(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): HFile for column family info-A not found Tests in error: test_TIMERANGE(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): /home/gchanan/workspace/apache92/target/test-data/276cbd0c-c771-4f81-9ba8-c464c9dd7486/test_TIMERANGE_present/_temporary/0/_temporary/_attempt_200707121733_0001_m_00_0 (Is a directory) testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable It looks like on trunk, this also results in an error: testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable I have a patch that fixes testColumnFamilyCompression and test_TIMERANGE, but haven't fixed the other 3 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5317) Fix TestHFileOutputFormat to work against hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200651#comment-13200651 ] Zhihong Yu commented on HBASE-5317: --- I found the following in test output (when using hadoop 0.23): {code} 2012-02-04 20:18:30,563 WARN [main] conf.Configuration(326): mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address2012-02-04 20:18:30,578 WARN [main] conf.Configuration(343): mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 2012-02-04 20:18:30,579 INFO [main] mapreduce.HFileOutputFormat(320): Writing partition information to /Users/zhihyu/trunk-hbase/target/test-data/67a9efc5-8b7c-4afd-b516-39dd454334b8/mapred-working-dir/partitions_864c11de-4d4f-45a3-ba94-26aac1c751e6 2012-02-04 20:18:30,604 WARN [IPC Server handler 4 on 63830] master.HMaster(1725): Failed getting all descriptors org.apache.hadoop.hbase.TableExistsException: No descriptor for target, hdfs://localhost:63821/user/zhihyu at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:164) at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:182) at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1723) {code} When using hadoop 1.0, there is no such error. Fix TestHFileOutputFormat to work against hadoop 0.23 - Key: HBASE-5317 URL: https://issues.apache.org/jira/browse/HBASE-5317 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0, 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Attachments: HBASE-5317-v0.patch Running mvn -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat yields this on 0.92: Failed tests: testColumnFamilyCompression(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): HFile for column family info-A not found Tests in error: test_TIMERANGE(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): /home/gchanan/workspace/apache92/target/test-data/276cbd0c-c771-4f81-9ba8-c464c9dd7486/test_TIMERANGE_present/_temporary/0/_temporary/_attempt_200707121733_0001_m_00_0 (Is a directory) testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable It looks like on trunk, this also results in an error: testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable I have a patch that fixes testColumnFamilyCompression and test_TIMERANGE, but haven't fixed the other 3 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5317) Fix TestHFileOutputFormat to work against hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200661#comment-13200661 ] Gregory Chanan commented on HBASE-5317: --- @Ted: Yes, that is the problem that causes the TestTable error. I'll post a patch for that tomorrow. Fix TestHFileOutputFormat to work against hadoop 0.23 - Key: HBASE-5317 URL: https://issues.apache.org/jira/browse/HBASE-5317 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0, 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Attachments: HBASE-5317-v0.patch Running mvn -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat yields this on 0.92: Failed tests: testColumnFamilyCompression(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): HFile for column family info-A not found Tests in error: test_TIMERANGE(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): /home/gchanan/workspace/apache92/target/test-data/276cbd0c-c771-4f81-9ba8-c464c9dd7486/test_TIMERANGE_present/_temporary/0/_temporary/_attempt_200707121733_0001_m_00_0 (Is a directory) testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable It looks like on trunk, this also results in an error: testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable I have a patch that fixes testColumnFamilyCompression and test_TIMERANGE, but haven't fixed the other 3 yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5337) AM.updateTimers() delays the timeout monitor from assigning regions.
AM.updateTimers() delays the timeout monitor from assigning regions. Key: HBASE-5337 URL: https://issues.apache.org/jira/browse/HBASE-5337 Project: HBase Issue Type: Bug Affects Versions: 0.90.5 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.90.7 AM.updateTimers() is necessary when the new RS has joined the cluster and there are more regions to be assigned to it. The updateTimer will help the TM from not picking up these regions because they are still in the process of assignment. But if in 0.90 we have 30min as TM period then this step of updating the timers further delays the assignment. I think we can remove the call for updateTimer in 0.90 till we decide to change the TM period. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5229) Explore building blocks for multi-row local transactions.
[ https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13200681#comment-13200681 ] jirapos...@reviews.apache.org commented on HBASE-5229: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3748/#review4822 --- A couple of nits and small implementation details, but overall looks pretty good. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java https://reviews.apache.org/r/3748/#comment10588 I think is this unnecessary, javadoc should handle inheriting the docs. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java https://reviews.apache.org/r/3748/#comment10587 or presplitting as is described in other documenttation. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java https://reviews.apache.org/r/3748/#comment10586 Probably want to wrap NOTE in b tags to call it out. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java https://reviews.apache.org/r/3748/#comment10590 A javadoc here might be nice to indicate that the nullary constructor is actually completely ok to use (as opposed to the more common state of being reserved for readFields). http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MultiRowMutation.java https://reviews.apache.org/r/3748/#comment10585 Even though it uses protected structures doesn't mean that its necessarily thread safe. In fact, because it is using the standard ArrayList, there is no guarantee of safety. Either the class should be marked as not thread safe OR the mutations should be wrapped as a concurrent list. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/RowMutation.java https://reviews.apache.org/r/3748/#comment10591 You really don't need to keep around the row anymore either because you can get that from the mutations as you already do mutateRows with MultiRowMutation. Its nice to store it, but is only going to be checked infrequently and saves you a little bit over the wire (which could add up, depending on row size). http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/3748/#comment10592 Suprised this isn't a utility method in HRegion - it seems really useful. Maybe worth pulling out for general use. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/3748/#comment10593 This isn't actually true, right? With multirow, you are actually going to lock more than one row (and the lockId null seems kind of a hack around that as it is always null, so far). http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/3748/#comment10594 nit: lockID rather than just lid would be slightly descriptive. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java https://reviews.apache.org/r/3748/#comment10595 But in the comments on the MultiRowMutation you push that checking off onto the RS, so no checking really happens then (except, I guess when you try to mutate rows on the region and it fails b/c those rows aren't there, but that seems kinda late for the check). http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java https://reviews.apache.org/r/3748/#comment10596 Wow, this is ugly. Maybe we should consider some refactoring of this later? http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java https://reviews.apache.org/r/3748/#comment10597 This class can get easily bloated as we add more types. Might be worth considering refactoring this out into its own test. - Jesse On 2012-02-03 19:59:55, Lars Hofhansl wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/3748/ bq. --- bq. bq. (Updated 2012-02-03 19:59:55) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This builds on HBASE-3584, HBASE-5203, and HBASE-5304. bq. bq. Multiple Rows can be locked and applied atomically as long as the application ensures that all rows reside in the same