[jira] [Commented] (HBASE-5353) HA/Distributed HMaster via RegionServers
[ https://issues.apache.org/jira/browse/HBASE-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225079#comment-13225079 ] Jonathan Hsieh commented on HBASE-5353: --- @Todd @Nicholas A months back I've started some work on HBASE-5083 but never haven't posted the patch yet since it has a pretty nasty hack in it (adding 10 to the master port to get the info/http port). There has been some cluster status things changing recently -- I'll wait for that to settle before I finish that patch. HA/Distributed HMaster via RegionServers Key: HBASE-5353 URL: https://issues.apache.org/jira/browse/HBASE-5353 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.94.0 Reporter: Jesse Yates Priority: Minor Currently, the HMaster node(s) must be considered a 'special' node (though not a single point of failover), meaning that the node must be protected more than the other cluster machines or at least specially monitored. Minimally, we always need to ensure that the master is running, rather than letting the system handle that internally. It should be possible to instead have the HMaster be much more available, either in a distributed sense (meaning a bit rewrite) or multiple, dynamically created instances combined with the hot fail-over of masters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5539) asynchbase PerformanceEvaluation
asynchbase PerformanceEvaluation
Key: HBASE-5539
URL: https://issues.apache.org/jira/browse/HBASE-5539
Project: HBase
Issue Type: New Feature
Components: performance
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Minor
I plugged [asynchbase|https://github.com/stumbleupon/asynchbase] into
{{PerformanceEvaluation}}. This enables testing asynchbase from
{{PerformanceEvaluation}} and comparing its performance to {{HTable}}. Also
asynchbase doesn't come with any benchmark, so it was good that I was able to
plug it into {{PerformanceEvaluation}} relatively easily.
I am in the processing of collecting results on a dev cluster running 0.92.1
and will publish them once they're ready.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5539) asynchbase PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-5539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoit Sigoure updated HBASE-5539:
--
Attachment: 0001-asynchbase-PerformanceEvaluation.patch
This is the patch that I'm using right now to run the tests. It currently
requires an unreleased version of asynchbase built from the tip of [my master
branch|https://github.com/tsuna/asynchbase]. Specifically, it's [this
change|https://github.com/tsuna/asynchbase/commit/06e619958f4ac8694c435d2da580e0ce77b43590]
that's required to help handle NSREs better in the mixed sync/async code of
{{PerformanceEvaluation}} + asynchbase. This code is not final and will change
as I'm still figuring out the best way to glue proper throttling during NSREs
in synchronous / semi-async code.
asynchbase PerformanceEvaluation
Key: HBASE-5539
URL: https://issues.apache.org/jira/browse/HBASE-5539
Project: HBase
Issue Type: New Feature
Components: performance
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Minor
Labels: benchmark
Attachments: 0001-asynchbase-PerformanceEvaluation.patch
I plugged [asynchbase|https://github.com/stumbleupon/asynchbase] into
{{PerformanceEvaluation}}. This enables testing asynchbase from
{{PerformanceEvaluation}} and comparing its performance to {{HTable}}. Also
asynchbase doesn't come with any benchmark, so it was good that I was able to
plug it into {{PerformanceEvaluation}} relatively easily.
I am in the processing of collecting results on a dev cluster running 0.92.1
and will publish them once they're ready.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5516) GZip leading to memory leak in 0.90. Fix similar to HBASE-5387 needed for 0.90.
[
https://issues.apache.org/jira/browse/HBASE-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225202#comment-13225202
]
ramkrishna.s.vasudevan commented on HBASE-5516:
---
{code}
top - 08:41:24 up 28 days, 17:46, 4 users, load average: 3.23, 2.73, 2.52
Tasks: 308 total, 1 running, 307 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.3%us, 2.4%sy, 0.0%ni, 78.8%id, 7.8%wa, 0.0%hi, 0.7%si, 0.0%st
Mem: 48264M total,48125M used, 139M free, 4370M buffers
Swap:51199M total, 53M used,51146M free,20926M cached
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
30934 root 20 0 23.2g 20g 12m S 144 42.6 314:25.74 java
30717 root 20 0 4859m 200m 10m S 50 0.4 66:09.54 java
7351 root 20 0 000 D2 0.0 14:30.49 kjournald
38 root 20 0 000 S1 0.0 0:36.31 events/3
127 root 20 0 000 S1 0.0 48:30.54 kswapd0
128 root 20 0 000 S1 0.0 47:48.75 kswapd1
4877 root 20 0 8968 400 260 S1 0.0 8:48.69 irqbalance
12644 root 20 0 8900 1356 852 R1 0.0 0:00.91 top
{code}
The above reports are before patch. You can see the native memory going beyond
20g.
{code}
{code}
GZip leading to memory leak in 0.90. Fix similar to HBASE-5387 needed for
0.90.
Key: HBASE-5516
URL: https://issues.apache.org/jira/browse/HBASE-5516
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.90.7
Attachments: HBASE-5516_2_0.90.patch
Usage of GZip is leading to resident memory leak in 0.90.
We need to have something similar to HBASE-5387 in 0.90.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5516) GZip leading to memory leak in 0.90. Fix similar to HBASE-5387 needed for 0.90.
[ https://issues.apache.org/jira/browse/HBASE-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5516: -- Attachment: HBASE-5516_3_0.90.patch Updated patch addressing comments. GZip leading to memory leak in 0.90. Fix similar to HBASE-5387 needed for 0.90. Key: HBASE-5516 URL: https://issues.apache.org/jira/browse/HBASE-5516 Project: HBase Issue Type: Bug Affects Versions: 0.90.5 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.90.7 Attachments: HBASE-5516_2_0.90.patch, HBASE-5516_3_0.90.patch Usage of GZip is leading to resident memory leak in 0.90. We need to have something similar to HBASE-5387 in 0.90. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5516) GZip leading to memory leak in 0.90. Fix similar to HBASE-5387 needed for 0.90.
[ https://issues.apache.org/jira/browse/HBASE-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225267#comment-13225267 ] ramkrishna.s.vasudevan commented on HBASE-5516: --- bq.What if blockBegin is 0 but less than HEADER_SIZE ? This will not happen Ted. GZip leading to memory leak in 0.90. Fix similar to HBASE-5387 needed for 0.90. Key: HBASE-5516 URL: https://issues.apache.org/jira/browse/HBASE-5516 Project: HBase Issue Type: Bug Affects Versions: 0.90.5 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.90.7 Attachments: HBASE-5516_2_0.90.patch, HBASE-5516_3_0.90.patch Usage of GZip is leading to resident memory leak in 0.90. We need to have something similar to HBASE-5387 in 0.90. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5505) if max file size set to 256M, Per table hbase.hregion.max.filesize setting(at htable creation) overwritten by value set in hbase-site.conf(which might not be 256MB)
[
https://issues.apache.org/jira/browse/HBASE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225275#comment-13225275
]
Jonathan Hsieh commented on HBASE-5505:
---
@steven I agree with your suggestion of where the value should be loaded. Do
you want to look into a fix for this?
if max file size set to 256M, Per table hbase.hregion.max.filesize setting(at
htable creation) overwritten by value set in hbase-site.conf(which might not
be 256MB)
Key: HBASE-5505
URL: https://issues.apache.org/jira/browse/HBASE-5505
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4, 0.92.0
Reporter: steven zhuang
seems to me that when set to DEFAULT_MAX_FILE_SIZE, the max region file size
set at Htable creation time may later be overwritten by value set in
hbase-site.conf, i.e. no Per table hbase.hregion.max.filesize at all.
in this case, on Web UI in the table description, max file is still 256M, but
we can find region file much bigger on HDFS.
In 0.92.0, o.a.h.hbase.regionserver.ConstantSizeRegionSplitPolicy, around
line 33:
long maxFileSize = region.getTableDesc().getMaxFileSize();
// By default we split region if a file
HConstants.DEFAULT_MAX_FILE_SIZE.
if (maxFileSize == HConstants.DEFAULT_MAX_FILE_SIZE) { //zx: here
setting is over written by configuaration from hbase-site.conf
maxFileSize = getConf().getLong(hbase.hregion.max.filesize,
HConstants.DEFAULT_MAX_FILE_SIZE);
}
In 0.90.4(cdh3u3) or earlier version, this is found in
o.a.h.hbase.regionserver.Store, around line 188:
long maxFileSize = info.getTableDesc().getMaxFileSize();
if (maxFileSize == HConstants.DEFAULT_MAX_FILE_SIZE) {
maxFileSize = conf.getLong(hbase.hregion.max.filesize,
HConstants.DEFAULT_MAX_FILE_SIZE);
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-5516) GZip leading to memory leak in 0.90. Fix similar to HBASE-5387 needed for 0.90.
[ https://issues.apache.org/jira/browse/HBASE-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225267#comment-13225267 ] ramkrishna.s.vasudevan edited comment on HBASE-5516 at 3/8/12 4:08 PM: --- bq.What if blockBegin is 0 but less than HEADER_SIZE ? This will not happen Ted. The new block will be created only after one block is completed. So the blcok begin should be 0 and not less than header size. Correct me if am wrong. was (Author: ram_krish): bq.What if blockBegin is 0 but less than HEADER_SIZE ? This will not happen Ted. GZip leading to memory leak in 0.90. Fix similar to HBASE-5387 needed for 0.90. Key: HBASE-5516 URL: https://issues.apache.org/jira/browse/HBASE-5516 Project: HBase Issue Type: Bug Affects Versions: 0.90.5 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.90.7 Attachments: HBASE-5516_2_0.90.patch, HBASE-5516_3_0.90.patch Usage of GZip is leading to resident memory leak in 0.90. We need to have something similar to HBASE-5387 in 0.90. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225278#comment-13225278 ] ramkrishna.s.vasudevan commented on HBASE-5074: --- @Dhruba Most of the times the 4 test cases keep failing. I think it should be ok. If TestMasterObserver is running locally then it should be fine i think. support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5540) Update apache jenkins to include 0.94 and builds against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5540: -- Description: Currently there is no hbase 0.94 apache jenkins build and the trunk on hadoop 0.23 builds are disabled. Ideally we should add the former and re-enable the latter. (was: Currently there is no hbase 0.94 apache jenkins build and the trunk on hadoop 0.23 builds are disabled. Ideally we should add the former and ren-enable the latter.) Update apache jenkins to include 0.94 and builds against Hadoop 0.23 Key: HBASE-5540 URL: https://issues.apache.org/jira/browse/HBASE-5540 Project: HBase Issue Type: Task Components: build, test Reporter: Jonathan Hsieh Labels: jenkins Currently there is no hbase 0.94 apache jenkins build and the trunk on hadoop 0.23 builds are disabled. Ideally we should add the former and re-enable the latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5540) Update apache jenkins to include 0.94 and builds against Hadoop 0.23
Update apache jenkins to include 0.94 and builds against Hadoop 0.23 Key: HBASE-5540 URL: https://issues.apache.org/jira/browse/HBASE-5540 Project: HBase Issue Type: Task Components: build, test Reporter: Jonathan Hsieh Currently there is no hbase 0.94 apache jenkins build and the trunk on hadoop 0.23 builds are disabled. Ideally we should add the former and ren-enable the latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225287#comment-13225287 ] Phabricator commented on HBASE-5515: sc has commented on the revision HBASE-5515 [jira] Add a processRow API that supports atomic multiple reads and writes on a row. @jyates: Thanks for the review comments. I think what you and Lars think makes sense. I guess I started this patch by using RowProcessor a parameter of HTable. So I feel RowProcessor should be a parameter. But with the Coprocessor case, it seems what you guys think makes more sense. I will update this patch soon. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/coprocessor/ProcessRowEndpoint.java:52 The abstract method approach is a good idea. Thanks! REVISION DETAIL https://reviews.facebook.net/D2067 Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5540) Update apache jenkins to include 0.94 and builds against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225300#comment-13225300 ] Zhihong Yu commented on HBASE-5540: --- There is 0.94 build: https://builds.apache.org/job/HBase-0.94/ Update apache jenkins to include 0.94 and builds against Hadoop 0.23 Key: HBASE-5540 URL: https://issues.apache.org/jira/browse/HBASE-5540 Project: HBase Issue Type: Task Components: build, test Reporter: Jonathan Hsieh Labels: jenkins Currently there is no hbase 0.94 apache jenkins build and the trunk on hadoop 0.23 builds are disabled. Ideally we should add the former and re-enable the latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5516) GZip leading to memory leak in 0.90. Fix similar to HBASE-5387 needed for 0.90.
[ https://issues.apache.org/jira/browse/HBASE-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225304#comment-13225304 ] Zhihong Yu commented on HBASE-5516: --- +1 on patch v3. GZip leading to memory leak in 0.90. Fix similar to HBASE-5387 needed for 0.90. Key: HBASE-5516 URL: https://issues.apache.org/jira/browse/HBASE-5516 Project: HBase Issue Type: Bug Affects Versions: 0.90.5 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.90.7 Attachments: HBASE-5516_2_0.90.patch, HBASE-5516_3_0.90.patch Usage of GZip is leading to resident memory leak in 0.90. We need to have something similar to HBASE-5387 in 0.90. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5540) Update apache jenkins to include 0.94 and builds against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225318#comment-13225318 ] Jonathan Hsieh commented on HBASE-5540: --- I see. In that case, we should update jenkins groups to make the build show up here with the others. https://builds.apache.org/view/G-L/view/HBase/ Sent from my iPhone Update apache jenkins to include 0.94 and builds against Hadoop 0.23 Key: HBASE-5540 URL: https://issues.apache.org/jira/browse/HBASE-5540 Project: HBase Issue Type: Task Components: build, test Reporter: Jonathan Hsieh Labels: jenkins Currently there is no hbase 0.94 apache jenkins build and the trunk on hadoop 0.23 builds are disabled. Ideally we should add the former and re-enable the latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5540) Update apache jenkins to include 0.94 and builds against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225322#comment-13225322 ] stack commented on HBASE-5540: -- Done. I missed adding it to the hbase group. Update apache jenkins to include 0.94 and builds against Hadoop 0.23 Key: HBASE-5540 URL: https://issues.apache.org/jira/browse/HBASE-5540 Project: HBase Issue Type: Task Components: build, test Reporter: Jonathan Hsieh Labels: jenkins Currently there is no hbase 0.94 apache jenkins build and the trunk on hadoop 0.23 builds are disabled. Ideally we should add the former and re-enable the latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5540) Update apache jenkins to include 0.94 and builds against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-5540. -- Resolution: Fixed Assignee: stack I grouped the 0.94 builds under the hbase area up on jenkins and reenabled the 0.23 builds. In all of these jobs, I've removed the lock on hbase. I did it originally in desperation to see if it was responsible for hbase builds hanging jenkins servers a while back. Since I did it, we've not had a hang. Hopefully this is it (though when I read the builds setup prescription, it suggests that long builds like ours should use a lock). Update apache jenkins to include 0.94 and builds against Hadoop 0.23 Key: HBASE-5540 URL: https://issues.apache.org/jira/browse/HBASE-5540 Project: HBase Issue Type: Task Components: build, test Reporter: Jonathan Hsieh Assignee: stack Labels: jenkins Currently there is no hbase 0.94 apache jenkins build and the trunk on hadoop 0.23 builds are disabled. Ideally we should add the former and re-enable the latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5515: --- Attachment: HBASE-5515.D2067.20.patch sc updated the revision HBASE-5515 [jira] Add a processRow API that supports atomic multiple reads and writes on a row. Reviewers: tedyu, dhruba, JIRA Addressed jyates's review comments, thanks. REVISION DETAIL https://reviews.facebook.net/D2067 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/coprocessor/RowProcessor.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestProcessRowEndpoint.java Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5508) Add an option to allow test output to show on the terminal
[ https://issues.apache.org/jira/browse/HBASE-5508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen resolved HBASE-5508. --- Resolution: Fixed Add an option to allow test output to show on the terminal -- Key: HBASE-5508 URL: https://issues.apache.org/jira/browse/HBASE-5508 Project: HBase Issue Type: Improvement Components: test Reporter: Scott Chen Assignee: Scott Chen Priority: Minor Fix For: 0.94.0, 0.96.0 Attachments: HBASE-5508.D2037.1.patch Sometimes it is useful to directly see the test results on the terminal. We can add a property to achieve that. mvn test -Dtest.output.tofile=false -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5540) Update apache jenkins to include 0.94 and builds against Hadoop 0.23
[ https://issues.apache.org/jira/browse/HBASE-5540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225332#comment-13225332 ] Jonathan Hsieh commented on HBASE-5540: --- Thanks Stack! Update apache jenkins to include 0.94 and builds against Hadoop 0.23 Key: HBASE-5540 URL: https://issues.apache.org/jira/browse/HBASE-5540 Project: HBase Issue Type: Task Components: build, test Reporter: Jonathan Hsieh Assignee: stack Labels: jenkins Currently there is no hbase 0.94 apache jenkins build and the trunk on hadoop 0.23 builds are disabled. Ideally we should add the former and re-enable the latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225333#comment-13225333 ] Phabricator commented on HBASE-5515: tedyu has commented on the revision HBASE-5515 [jira] Add a processRow API that supports atomic multiple reads and writes on a row. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/coprocessor/RowProcessor.java:53 Is abstract keyword needed ? REVISION DETAIL https://reviews.facebook.net/D2067 Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[
https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225340#comment-13225340
]
Phabricator commented on HBASE-5515:
sc has commented on the revision HBASE-5515 [jira] Add a processRow API that
supports atomic multiple reads and writes on a row.
I tried making ProcessorRowEnpoint endpoint an abstract class. The difficulty
is that when creating ProcessRowProtocal you can not pass parameter in the
constructor. So the parameter has to be passed by the actually processRow()
call. You can use generic
ProcessRowProtocol {
T, S T processRow(S input);
}
But this is kind of ugly. And you will need to have the actual S in the class
path also. This is not good for the same reason you don't what RowProcessor as
a separate class.
So I just leave the users to do their own CoprocessorProtocol and
CoprocessorEndpoint just like the example in the unit test.
REVISION DETAIL
https://reviews.facebook.net/D2067
Add a processRow API that supports atomic multiple reads and writes on a row
Key: HBASE-5515
URL: https://issues.apache.org/jira/browse/HBASE-5515
Project: HBase
Issue Type: New Feature
Reporter: Scott Chen
Assignee: Scott Chen
Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch,
HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch,
HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch,
HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch,
HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch,
HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch,
HBASE-5515.D2067.20.patch, HBASE-5515.D2067.3.patch,
HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch,
HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch
We have modified HRegion.java internally to do some atomic row processing. It
will be nice to have a plugable API for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225339#comment-13225339 ] Phabricator commented on HBASE-5515: sc has commented on the revision HBASE-5515 [jira] Add a processRow API that supports atomic multiple reads and writes on a row. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/coprocessor/RowProcessor.java:53 goodcatch REVISION DETAIL https://reviews.facebook.net/D2067 Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5514) Compile against hadoop 0.24-SNAPSHOT
[ https://issues.apache.org/jira/browse/HBASE-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225338#comment-13225338 ] Zhihong Yu commented on HBASE-5514: --- TestSplitLogManager passed locally based on patch v4. Integrated to TRUNK. Thanks for the patch, Mingjie. Compile against hadoop 0.24-SNAPSHOT Key: HBASE-5514 URL: https://issues.apache.org/jira/browse/HBASE-5514 Project: HBase Issue Type: Bug Components: build, test Affects Versions: 0.92.0, 0.94.0 Reporter: Mingjie Lai Assignee: Mingjie Lai Priority: Minor Fix For: 0.94.0 Attachments: HBASE-5514-2.patch, HBASE-5514-3.patch, HBASE-5514-4.patch, HBASE-5514.patch Need to compile hbase against the latest hadoop trunk which just had NN HA merged in. 1) add a hadoop 0.24 profile 2) HBASE-5480 3) HADOOP-8124 removed deprecated Syncable.sync(). It brings compile errors for hbase against hadoop trunk(0.24). TestHLogSplit and TestHLog still call the deprecated sync(). Need to replace it with hflush() so the compilation can pass. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5515: --- Attachment: HBASE-5515.D2067.21.patch sc updated the revision HBASE-5515 [jira] Add a processRow API that supports atomic multiple reads and writes on a row. Reviewers: tedyu, dhruba, JIRA Remove the redundant abstract keyword REVISION DETAIL https://reviews.facebook.net/D2067 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/coprocessor/RowProcessor.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestProcessRowEndpoint.java Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225345#comment-13225345 ] Phabricator commented on HBASE-5515: tedyu has accepted the revision HBASE-5515 [jira] Add a processRow API that supports atomic multiple reads and writes on a row. If tests pass. REVISION DETAIL https://reviews.facebook.net/D2067 BRANCH 5515 Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5515: --- Attachment: HBASE-5515.D2067.22.patch sc updated the revision HBASE-5515 [jira] Add a processRow API that supports atomic multiple reads and writes on a row. Reviewers: tedyu, dhruba, JIRA Fixed the indentation and remove an unused method REVISION DETAIL https://reviews.facebook.net/D2067 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/coprocessor/RowProcessor.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestProcessRowEndpoint.java Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225364#comment-13225364 ] Hadoop QA commented on HBASE-5515: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517577/HBASE-5515.D2067.20.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -129 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1133//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1133//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1133//console This message is automatically generated. Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks Key: HBASE-5541 URL: https://issues.apache.org/jira/browse/HBASE-5541 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed. Similar to what we do in doMiniBatchPut, we should create the log entry with the lock held, but only sync the HLog after the log is released, along with rollback logic in case the sync'ing fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225374#comment-13225374 ] jirapos...@reviews.apache.org commented on HBASE-5526: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4217/#review5738 --- Ship it! One minor comment and some whitespace (can fix upon commit). src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/4217/#comment12489 We don't log in the HFile case above. I guess we log either in both cases or not at all. LOG.debug in both cases seems fine to me. - Lars On 2012-03-08 18:31:13, Jesse Yates wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4217/ bq. --- bq. bq. (Updated 2012-03-08 18:31:13) bq. bq. bq. Review request for hbase and Lars Hofhansl. bq. bq. bq. Summary bq. --- bq. bq. Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. bq. bq. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. bq. bq. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. bq. bq. bq. This addresses bug HBASE-5526. bq. https://issues.apache.org/jira/browse/HBASE-5526 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/HConstants.java e60ce04 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 9e7e624 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 76ff422 bq.src/main/java/org/apache/hadoop/hbase/util/FSUtils.java d2d7efe bq.src/main/resources/hbase-default.xml 9277e0c bq.src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java e2611e6 bq. bq. Diff: https://reviews.apache.org/r/4217/diff bq. bq. bq. Testing bq. --- bq. bq. mvn clean test -P localTests passes bq. bq. bq. Thanks, bq. bq. Jesse bq. bq. Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5521) Move compression/decompression to an encoder specific encoding context
[ https://issues.apache.org/jira/browse/HBASE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225379#comment-13225379 ] He Yongqiang commented on HBASE-5521: - Tests passed in my local test, not sure why failed on Jenkins. Move compression/decompression to an encoder specific encoding context -- Key: HBASE-5521 URL: https://issues.apache.org/jira/browse/HBASE-5521 Project: HBase Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Attachments: HBASE-5521.1.patch, HBASE-5521.D2097.1.patch, HBASE-5521.D2097.2.patch, HBASE-5521.D2097.3.patch, HBASE-5521.D2097.4.patch, HBASE-5521.D2097.5.patch As part of working on HBASE-5313, we want to add a new columnar encoder/decoder. It makes sense to move compression to be part of encoder/decoder: 1) a scanner for a columnar encoded block can do lazy decompression to a specific part of a key value object 2) avoid an extra bytes copy from encoder to hblock-writer. If there is no encoder specified for a writer, the HBlock.Writer will use a default compression-context to do something very similar to today's code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225392#comment-13225392 ] Hadoop QA commented on HBASE-5515: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517582/HBASE-5515.D2067.21.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -129 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplicationPeer org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1134//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1134//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1134//console This message is automatically generated. Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225403#comment-13225403 ] Hadoop QA commented on HBASE-5515: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517584/HBASE-5515.D2067.22.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -129 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1135//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1135//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1135//console This message is automatically generated. Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
[ https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225407#comment-13225407 ] Zhihong Yu commented on HBASE-5541: --- bq. after the log is released I guess you meant 'after the lock is released' Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks Key: HBASE-5541 URL: https://issues.apache.org/jira/browse/HBASE-5541 Project: HBase Issue Type: Sub-task Components: client, regionserver Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed. Similar to what we do in doMiniBatchPut, we should create the log entry with the lock held, but only sync the HLog after the log is released, along with rollback logic in case the sync'ing fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
[ https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5541: - Description: Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed. Similar to what we do in doMiniBatchPut, we should create the log entry with the lock held, but only sync the HLog after the lock is released, along with rollback logic in case the sync'ing fails. was: Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed. Similar to what we do in doMiniBatchPut, we should create the log entry with the lock held, but only sync the HLog after the log is released, along with rollback logic in case the sync'ing fails. Indeed. Fixed. Thanks for pointing out Ted. Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks Key: HBASE-5541 URL: https://issues.apache.org/jira/browse/HBASE-5541 Project: HBase Issue Type: Sub-task Components: client, regionserver Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed. Similar to what we do in doMiniBatchPut, we should create the log entry with the lock held, but only sync the HLog after the lock is released, along with rollback logic in case the sync'ing fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225417#comment-13225417 ] Phabricator commented on HBASE-5515: lhofhansl has commented on the revision HBASE-5515 [jira] Add a processRow API that supports atomic multiple reads and writes on a row. +1... Ship it. We should unify HRegion.mutateRowWithLocks and HRegion.processRow, but let's do that as a separate issue. REVISION DETAIL https://reviews.facebook.net/D2067 BRANCH 5515 Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5515: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517326/HBASE-5515.D2067.18.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -129 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1123//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1123//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1123//console This message is automatically generated.) Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5515: -- Hadoop Flags: Reviewed @Lars: Do you think this feature should go to 0.94 ? Please assign 'Fix Version' accordingly. Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5515: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517313/HBASE-5515.D2067.17.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -129 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1119//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1119//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1119//console This message is automatically generated.) Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
Unify HRegion.mutateRowsWithLocks() and HRegion.processRow() Key: HBASE-5542 URL: https://issues.apache.org/jira/browse/HBASE-5542 Project: HBase Issue Type: Improvement Reporter: Scott Chen Assignee: Scott Chen mutateRowsWithLocks() does atomic mutations on multiple rows. processRow() does atomic read-modify-writes on a single row. It will be useful to generalize both and have a processRowsWithLocks() that does atomic read-modify-writes on multiple rows. This also helps reduce some redundancy in the codes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225430#comment-13225430 ] Phabricator commented on HBASE-5515: sc has commented on the revision HBASE-5515 [jira] Add a processRow API that supports atomic multiple reads and writes on a row. @Lars, @Ted, @Jesse: Thanks for the reviews :) You guys helped a lot. I created https://issues.apache.org/jira/browse/HBASE-5542 for the unify work. REVISION DETAIL https://reviews.facebook.net/D2067 BRANCH 5515 Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225433#comment-13225433 ] Zhihong Yu commented on HBASE-5515: --- Integrated to TRUNK. Thanks for the patch, Scott. Thanks for the review Lars and Jesse. Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5515: - Fix Version/s: 0.96.0 I think this should go into 0.96. We can backport later if the need arises. Scott, what do you think? Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.96.0 Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225469#comment-13225469 ] dhruba borthakur commented on HBASE-5515: - My vote is to put it into hbase-trunk only (and not to backport to 0.94) Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.96.0 Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row
[ https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225476#comment-13225476 ] Lars Hofhansl commented on HBASE-5515: -- Yeah with 0.96 above I mean trunk. Add a processRow API that supports atomic multiple reads and writes on a row Key: HBASE-5515 URL: https://issues.apache.org/jira/browse/HBASE-5515 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.96.0 Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch We have modified HRegion.java internally to do some atomic row processing. It will be nice to have a plugable API for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
[ https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5541: - Attachment: 5541.txt Here's a patch. TestAtomicOperation still passes fine. Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks Key: HBASE-5541 URL: https://issues.apache.org/jira/browse/HBASE-5541 Project: HBase Issue Type: Sub-task Components: client, regionserver Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 5541.txt Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed. Similar to what we do in doMiniBatchPut, we should create the log entry with the lock held, but only sync the HLog after the lock is released, along with rollback logic in case the sync'ing fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
[ https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5541: - Status: Patch Available (was: Open) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks Key: HBASE-5541 URL: https://issues.apache.org/jira/browse/HBASE-5541 Project: HBase Issue Type: Sub-task Components: client, regionserver Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 5541.txt Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed. Similar to what we do in doMiniBatchPut, we should create the log entry with the lock held, but only sync the HLog after the lock is released, along with rollback logic in case the sync'ing fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5480) Fixups to MultithreadedTableMapper for Hadoop 0.23.2+
[ https://issues.apache.org/jira/browse/HBASE-5480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225482#comment-13225482 ] Lars Hofhansl commented on HBASE-5480: -- But this should go into 0.94, no? Fixups to MultithreadedTableMapper for Hadoop 0.23.2+ - Key: HBASE-5480 URL: https://issues.apache.org/jira/browse/HBASE-5480 Project: HBase Issue Type: Bug Components: mapreduce Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Critical Fix For: 0.94.0 Attachments: HBASE-5480.patch There are two issues: - StatusReporter has a new method getProgress() - Mapper and reducer context objects can no longer be directly instantiated. See attached patch. I'm not thrilled with the added reflection but it was the minimally intrusive change. Raised the priority to critical because compilation fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5529) MR test failures becuase MALLOC_ARENA_MAX is not set
[ https://issues.apache.org/jira/browse/HBASE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5529: - Fix Version/s: 0.96.0 0.94.0 0.92.1 @Stack: Is it 0.92.1 now or 0.92.2? MR test failures becuase MALLOC_ARENA_MAX is not set Key: HBASE-5529 URL: https://issues.apache.org/jira/browse/HBASE-5529 Project: HBase Issue Type: Bug Components: mapreduce, test Affects Versions: 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor Fix For: 0.92.1, 0.94.0, 0.96.0 Attachments: HBASE-5529-to92.patch, HBASE-5529-trunk.patch When running unit tests on CentOS 6 I get a bunch of unit test failures in mapreduce-related tests due to: 2012-03-03 00:14:18,776 WARN [Container Monitor] monitor.ContainersMonitorImpl$MonitoringThread(436): Container [pid=21446,containerID=container_1330762435849_0002_01_01] is running beyond virtual memory limits. Current usage: 223.1mb of 2.0gb physical memory used; 6.9gb of 4.2gb virtual memory used. Killing container. Note: this also came up in the mapreduce project. See: https://issues.apache.org/jira/browse/MAPREDUCE-3933 Patch coming shortly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5543) Add a keepalive option for IPC connections
Add a keepalive option for IPC connections
--
Key: HBASE-5543
URL: https://issues.apache.org/jira/browse/HBASE-5543
Project: HBase
Issue Type: Improvement
Components: client, coprocessors, ipc
Reporter: Andrew Purtell
On the user list someone wrote in with a connection failure due to a long
running coprocessor:
{quote}
On Wed, Mar 7, 2012 at 10:59 PM, raghavendhra rahul wrote:
2012-03-08 12:03:09,475 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call execCoprocessor([B@50cb21, getProjection(), rpc version=1,
client version=0, methodsFingerPrint=0), rpc version=1, client version=29,
methodsFingerPrint=54742778 from 10.184.17.26:46472: output error
2012-03-08 12:03:09,476 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020 caught: java.nio.channels.ClosedChannelException
{quote}
I suggested in response we might consider give our RPC a keepalive option for
calls that may run for a long time (like execCoprocessor).
LarsH +1ed the idea:
{quote}
+1 on keepalive. It's a shame (especially for long running server code) to do
all the work, just to find out at the end that the client has given up.
Or maybe there should be a way to cancel an operation if the clients decides it
does not want to wait any longer (PostgreSQL does that for example). Here that
would mean the server would need to check periodically and coprocessors would
need to be written to support that - so maybe that's no-starter.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5529) MR test failures becuase MALLOC_ARENA_MAX is not set
[ https://issues.apache.org/jira/browse/HBASE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225491#comment-13225491 ] Lars Hofhansl commented on HBASE-5529: -- Committed to 0.94 and trunk. Waiting for stack on 0.92 MR test failures becuase MALLOC_ARENA_MAX is not set Key: HBASE-5529 URL: https://issues.apache.org/jira/browse/HBASE-5529 Project: HBase Issue Type: Bug Components: mapreduce, test Affects Versions: 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor Fix For: 0.92.1, 0.94.0, 0.96.0 Attachments: HBASE-5529-to92.patch, HBASE-5529-trunk.patch When running unit tests on CentOS 6 I get a bunch of unit test failures in mapreduce-related tests due to: 2012-03-03 00:14:18,776 WARN [Container Monitor] monitor.ContainersMonitorImpl$MonitoringThread(436): Container [pid=21446,containerID=container_1330762435849_0002_01_01] is running beyond virtual memory limits. Current usage: 223.1mb of 2.0gb physical memory used; 6.9gb of 4.2gb virtual memory used. Killing container. Note: this also came up in the mapreduce project. See: https://issues.apache.org/jira/browse/MAPREDUCE-3933 Patch coming shortly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
[
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225492#comment-13225492
]
Zhihong Yu commented on HBASE-5541:
---
There're two step 11 now:
{code}
+if (acquiredLocks != null) {
+ // 11. release the row lock
{code}
Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
Key: HBASE-5541
URL: https://issues.apache.org/jira/browse/HBASE-5541
Project: HBase
Issue Type: Sub-task
Components: client, regionserver
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Fix For: 0.94.0
Attachments: 5541.txt
Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
Similar to what we do in doMiniBatchPut, we should create the log entry with
the lock held, but only sync the HLog after the lock is released, along with
rollback logic in case the sync'ing fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
[ https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225503#comment-13225503 ] Lars Hofhansl commented on HBASE-5541: -- Whoops. Yes, that comment should've been removed. Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks Key: HBASE-5541 URL: https://issues.apache.org/jira/browse/HBASE-5541 Project: HBase Issue Type: Sub-task Components: client, regionserver Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 5541.txt Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed. Similar to what we do in doMiniBatchPut, we should create the log entry with the lock held, but only sync the HLog after the lock is released, along with rollback logic in case the sync'ing fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225506#comment-13225506 ] jirapos...@reviews.apache.org commented on HBASE-5526: -- bq. On 2012-03-08 18:39:02, Lars Hofhansl wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 770 bq. https://reviews.apache.org/r/4217/diff/3/?file=89166#file89166line770 bq. bq. We don't log in the HFile case above. bq. I guess we log either in both cases or not at all. bq. LOG.debug in both cases seems fine to me. Prefer to drop this as we already have the debug comment in FSUtils.create for the file and perms being created. This is just excessive logging. I'll post a new version and try to get most of the whitespace issues on the original ticket. - Jesse --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4217/#review5738 --- On 2012-03-08 18:31:13, Jesse Yates wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4217/ bq. --- bq. bq. (Updated 2012-03-08 18:31:13) bq. bq. bq. Review request for hbase and Lars Hofhansl. bq. bq. bq. Summary bq. --- bq. bq. Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. bq. bq. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. bq. bq. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. bq. bq. bq. This addresses bug HBASE-5526. bq. https://issues.apache.org/jira/browse/HBASE-5526 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/HConstants.java e60ce04 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 9e7e624 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 76ff422 bq.src/main/java/org/apache/hadoop/hbase/util/FSUtils.java d2d7efe bq.src/main/resources/hbase-default.xml 9277e0c bq.src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java e2611e6 bq. bq. Diff: https://reviews.apache.org/r/4217/diff bq. bq. bq. Testing bq. --- bq. bq. mvn clean test -P localTests passes bq. bq. bq. Thanks, bq. bq. Jesse bq. bq. Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5522) hbase 0.92 test artifacts are missing from Maven central
[ https://issues.apache.org/jira/browse/HBASE-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225509#comment-13225509 ] Hudson commented on HBASE-5522: --- Integrated in HBase-0.92-security #97 (See [https://builds.apache.org/job/HBase-0.92-security/97/]) HBASE-5522 hbase 0.92 test artifacts are missing from Maven central (Revision 1297684) HBASE-5522 hbase 0.92 test artifacts are missing from Maven central (Revision 1297683) Result = FAILURE stack : Files : * /hbase/branches/0.92/pom.xml stack : Files : * /hbase/branches/0.92/CHANGES.txt hbase 0.92 test artifacts are missing from Maven central Key: HBASE-5522 URL: https://issues.apache.org/jira/browse/HBASE-5522 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.92.0 Reporter: Roman Shaposhnik Assignee: stack Fix For: 0.92.1, 0.94.0 Attachments: 5522.txt Could someone with enough karma, please, publish the test artifacts for 0.92.0? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4890) fix possible NPE in HConnectionManager
[
https://issues.apache.org/jira/browse/HBASE-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225511#comment-13225511
]
Hudson commented on HBASE-4890:
---
Integrated in HBase-0.92-security #97 (See
[https://builds.apache.org/job/HBase-0.92-security/97/])
HBASE-4890 fix possible NPE in HConnectionManager (Revision 1298270)
Result = FAILURE
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
fix possible NPE in HConnectionManager
--
Key: HBASE-4890
URL: https://issues.apache.org/jira/browse/HBASE-4890
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jonathan Hsieh
Assignee: stack
Priority: Blocker
Fix For: 0.92.1, 0.94.0
Attachments: 4890-v2.txt, 4890-v3.txt, 4890-v3.txt, 4890-v3.txt,
4890.txt, splits.txt
I was running YCSB against a 0.92 branch and encountered this error message:
{code}
11/11/29 08:47:16 WARN client.HConnectionManager$HConnectionImplementation:
Failed all from
region=usertable,user3917479014967760871,1322555655231.f78d161e5724495a9723bcd972f97f41.,
hostname=c0316.hal.cloudera.com, port=57020
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1501)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1353)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:898)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:775)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:750)
at com.yahoo.ycsb.db.HBaseClient.update(Unknown Source)
at com.yahoo.ycsb.DBWrapper.update(Unknown Source)
at com.yahoo.ycsb.workloads.CoreWorkload.doTransactionUpdate(Unknown
Source)
at com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(Unknown Source)
at com.yahoo.ycsb.ClientThread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1315)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1327)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1325)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:158)
at $Proxy4.multi(Unknown Source)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1330)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1328)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1309)
... 7 more
{code}
It looks like the NPE is caused by server being null in the MultiRespone
call() method.
{code}
public MultiResponse call() throws IOException {
return getRegionServerWithoutRetries(
new ServerCallableMultiResponse(connection, tableName, null) {
public MultiResponse call() throws IOException {
return server.multi(multi);
}
@Override
public void connect(boolean reload) throws IOException {
server =
connection.getHRegionConnection(loc.getHostname(),
loc.getPort());
}
}
);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5531) Maven hadoop profile (version 23) needs to be updated with latest 23 snapshot
[ https://issues.apache.org/jira/browse/HBASE-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225513#comment-13225513 ] Hudson commented on HBASE-5531: --- Integrated in HBase-0.92-security #97 (See [https://builds.apache.org/job/HBase-0.92-security/97/]) HBASE-5531 Maven hadoop profile (version 23) needs to be updated with latest 23 snapshot (Laxman) (Revision 1297589) Result = FAILURE ramkrishna : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/pom.xml Maven hadoop profile (version 23) needs to be updated with latest 23 snapshot - Key: HBASE-5531 URL: https://issues.apache.org/jira/browse/HBASE-5531 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.92.2 Reporter: Laxman Assignee: Laxman Labels: build Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: HBASE-5531-trunk.patch, HBASE-5531.patch Current profile is still pointing to 0.23.1-SNAPSHOT. This is failing to build as 23.1 is already released and snapshot is not available anymore. We can update this to 0.23.2-SNAPSHOT. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5519) Incorrect warning in splitlogmanager
[ https://issues.apache.org/jira/browse/HBASE-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225514#comment-13225514 ] Hudson commented on HBASE-5519: --- Integrated in HBase-0.92-security #97 (See [https://builds.apache.org/job/HBase-0.92-security/97/]) HBASE-5519 Incorrect warning in splitlogmanager (Revision 1297710) Result = FAILURE stack : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java Incorrect warning in splitlogmanager Key: HBASE-5519 URL: https://issues.apache.org/jira/browse/HBASE-5519 Project: HBase Issue Type: Improvement Reporter: Prakash Khemani Assignee: Prakash Khemani Attachments: 0001-HBASE-5519-Incorrect-warning-in-splitlogmanager.patch, 5519.92.txt because of recently added behavior - where the splitlogmanager timeout thread get's data from zk node just to check that the zk node is there ... we might have multiple watches firing without the task znode expiring. remove the poor warning message. (internally, there was an assert that failed in Mikhail's tests) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5524) Add a couple of more filters to our rat exclusion set
[ https://issues.apache.org/jira/browse/HBASE-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225510#comment-13225510 ] Hudson commented on HBASE-5524: --- Integrated in HBase-0.92-security #97 (See [https://builds.apache.org/job/HBase-0.92-security/97/]) HBASE-5524 Add a couple of more filters to our rat exclusion set (Revision 1297279) Result = FAILURE stack : Files : * /hbase/branches/0.92/pom.xml Add a couple of more filters to our rat exclusion set - Key: HBASE-5524 URL: https://issues.apache.org/jira/browse/HBASE-5524 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.92.1, 0.94.0 Attachments: rat.txt Build up on jenkins is failing because I just enabled the rat/license check as part of our build. We're failing because CP is writing test data into top-level at ./test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5537) MXBean shouldn't have a dependence on InterfaceStability until 0.96
[ https://issues.apache.org/jira/browse/HBASE-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225515#comment-13225515 ] Hudson commented on HBASE-5537: --- Integrated in HBase-0.92-security #97 (See [https://builds.apache.org/job/HBase-0.92-security/97/]) HBASE-5537 MXBean shouldn't have a dependence on InterfaceStability until 0.96 (Revision 1298037) Result = FAILURE stack : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/MXBean.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/MXBean.java MXBean shouldn't have a dependence on InterfaceStability until 0.96 --- Key: HBASE-5537 URL: https://issues.apache.org/jira/browse/HBASE-5537 Project: HBase Issue Type: Bug Reporter: Jean-Daniel Cryans Assignee: stack Fix For: 0.92.1, 0.94.0 Attachments: 5537.txt HBASE-5325 has a dependence on InterfaceStability.Evolving in 0.92 and it shouldn't have it until 0.96. One problem it currently causes is that 0.92 can't be compiled against CDH3u3. Assigning to Stack. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5526: --- Status: Patch Available (was: Open) Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5526: --- Attachment: java_HBASE-5526-v5.patch Applying fixes from RB - should be ready to go. Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5529) MR test failures becuase MALLOC_ARENA_MAX is not set
[ https://issues.apache.org/jira/browse/HBASE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225522#comment-13225522 ] Lars Hofhansl commented on HBASE-5529: -- NM. Checked another jira that Stack committed yesterday. Was targeted at 0.92.1. MR test failures becuase MALLOC_ARENA_MAX is not set Key: HBASE-5529 URL: https://issues.apache.org/jira/browse/HBASE-5529 Project: HBase Issue Type: Bug Components: mapreduce, test Affects Versions: 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor Fix For: 0.92.1, 0.94.0, 0.96.0 Attachments: HBASE-5529-to92.patch, HBASE-5529-trunk.patch When running unit tests on CentOS 6 I get a bunch of unit test failures in mapreduce-related tests due to: 2012-03-03 00:14:18,776 WARN [Container Monitor] monitor.ContainersMonitorImpl$MonitoringThread(436): Container [pid=21446,containerID=container_1330762435849_0002_01_01] is running beyond virtual memory limits. Current usage: 223.1mb of 2.0gb physical memory used; 6.9gb of 4.2gb virtual memory used. Killing container. Note: this also came up in the mapreduce project. See: https://issues.apache.org/jira/browse/MAPREDUCE-3933 Patch coming shortly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5529) MR test failures becuase MALLOC_ARENA_MAX is not set
[ https://issues.apache.org/jira/browse/HBASE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5529: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) MR test failures becuase MALLOC_ARENA_MAX is not set Key: HBASE-5529 URL: https://issues.apache.org/jira/browse/HBASE-5529 Project: HBase Issue Type: Bug Components: mapreduce, test Affects Versions: 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor Fix For: 0.92.1, 0.94.0, 0.96.0 Attachments: HBASE-5529-to92.patch, HBASE-5529-trunk.patch When running unit tests on CentOS 6 I get a bunch of unit test failures in mapreduce-related tests due to: 2012-03-03 00:14:18,776 WARN [Container Monitor] monitor.ContainersMonitorImpl$MonitoringThread(436): Container [pid=21446,containerID=container_1330762435849_0002_01_01] is running beyond virtual memory limits. Current usage: 223.1mb of 2.0gb physical memory used; 6.9gb of 4.2gb virtual memory used. Killing container. Note: this also came up in the mapreduce project. See: https://issues.apache.org/jira/browse/MAPREDUCE-3933 Patch coming shortly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225528#comment-13225528 ] Zhihong Yu commented on HBASE-5074: --- I ran TestMasterObserver 5 times and it passed. TestReplicationPeer fails easily with or without the patch. Failures for TestTableMapReduce and TestHFileOutputFormat should be fixed by MAPREDUCE-3583 support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225529#comment-13225529 ] Lars Hofhansl commented on HBASE-5526: -- +1 Will commit soon (if tests pass) and nobody objects. Jesse, did you validate that this satisfies our requirements for the JSPs? Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225535#comment-13225535 ] Hadoop QA commented on HBASE-5526: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517610/java_HBASE-5526-v5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1137//console This message is automatically generated. Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5509) MR based copier for copying HFiles (trunk version)
[ https://issues.apache.org/jira/browse/HBASE-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225539#comment-13225539 ] Lars Hofhansl commented on HBASE-5509: -- Any takers for a review? I assume +1 from the FB guys (right Karthik?) :) MR based copier for copying HFiles (trunk version) -- Key: HBASE-5509 URL: https://issues.apache.org/jira/browse/HBASE-5509 Project: HBase Issue Type: Sub-task Components: documentation, regionserver Reporter: Karthik Ranganathan Assignee: Lars Hofhansl Fix For: 0.94.0, 0.96.0 Attachments: 5509-v2.txt, 5509.txt This copier is a modification of the distcp tool in HDFS. It does the following: 1. List out all the regions in the HBase cluster for the required table 2. Write the above out to a file 3. Each mapper 3.1 lists all the HFiles for a given region by querying the regionserver 3.2 copies all the HFiles 3.3 outputs success if the copy succeeded, failure otherwise. Failed regions are retried in another loop 4. Mappers are placed on nodes which have maximum locality for a given region to speed up copying -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225541#comment-13225541 ] Jesse Yates commented on HBASE-5526: Yup, works for blocking out the hfiles and .regioninfo. Odd that it didn't apply. Going to try rebasing and see if that was the problem. Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5526) Configurable file and directory based umask
[
https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225546#comment-13225546
]
Lars Hofhansl commented on HBASE-5526:
--
{code}
patching file src/main/java/org/apache/hadoop/hbase/HConstants.java
Hunk #1 succeeded at 610 (offset 4 lines).
patching file
src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
patching file src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Hunk #1 succeeded at 67 (offset 2 lines).
Hunk #2 FAILED at 79.
Hunk #3 FAILED at 89.
Hunk #4 succeeded at 342 (offset 4 lines).
Hunk #5 succeeded at 770 (offset 8 lines).
2 out of 5 hunks FAILED -- saving rejects to file
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java.rej
patching file src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
patching file src/main/resources/hbase-default.xml
patching file src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java
PATCH APPLICATION FAILED
{code}
HRegion was just recently changed.
Configurable file and directory based umask
---
Key: HBASE-5526
URL: https://issues.apache.org/jira/browse/HBASE-5526
Project: HBase
Issue Type: New Feature
Components: regionserver
Reporter: Jesse Yates
Assignee: Jesse Yates
Fix For: 0.94.0
Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch,
java_HBASE-5526-v5.patch, java_HBASE-5526.patch
Currently many all the files created by the HBase user are just written using
the default file permissions granted by hdfs. However, to ensure only the
correct user/group views the files and directories, we need to be able to
apply a configurable umask to either directories or files.
This ticket covers setting permissions for files written to dfs, as opposed
to things like pid and log files.
The impetus for this was to allow the web-user to view the directory
structure of hbase, but not to actually see any of the actual data hbase is
storing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5526: --- Attachment: java_HBASE-5526-v7.patch Rebased against 0.94. I believe this also applies against trunk, if we want to commit it there as well. Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5509) MR based copier for copying HFiles (trunk version)
[ https://issues.apache.org/jira/browse/HBASE-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225559#comment-13225559 ] Jesse Yates commented on HBASE-5509: Looking at it today - should have my comments in by COB. MR based copier for copying HFiles (trunk version) -- Key: HBASE-5509 URL: https://issues.apache.org/jira/browse/HBASE-5509 Project: HBase Issue Type: Sub-task Components: documentation, regionserver Reporter: Karthik Ranganathan Assignee: Lars Hofhansl Fix For: 0.94.0, 0.96.0 Attachments: 5509-v2.txt, 5509.txt This copier is a modification of the distcp tool in HDFS. It does the following: 1. List out all the regions in the HBase cluster for the required table 2. Write the above out to a file 3. Each mapper 3.1 lists all the HFiles for a given region by querying the regionserver 3.2 copies all the HFiles 3.3 outputs success if the copy succeeded, failure otherwise. Failed regions are retried in another loop 4. Mappers are placed on nodes which have maximum locality for a given region to speed up copying -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5538) A metric to measure the size of the response queue in the hbase rpc server
[ https://issues.apache.org/jira/browse/HBASE-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5538: - Status: Patch Available (was: Open) A metric to measure the size of the response queue in the hbase rpc server -- Key: HBASE-5538 URL: https://issues.apache.org/jira/browse/HBASE-5538 Project: HBase Issue Type: Improvement Components: ipc Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: D2199.1.patch The HbaseServer queues responses to client (if the client is slow). Expose a metric that records the size of the response queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5538) A metric to measure the size of the response queue in the hbase rpc server
[ https://issues.apache.org/jira/browse/HBASE-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225562#comment-13225562 ] Phabricator commented on HBASE-5538: stack has accepted the revision [jira] [HBASE-5538] A metric to measure the size of the response queue in the hbase rpc server. +1 REVISION DETAIL https://reviews.facebook.net/D2199 BRANCH svn A metric to measure the size of the response queue in the hbase rpc server -- Key: HBASE-5538 URL: https://issues.apache.org/jira/browse/HBASE-5538 Project: HBase Issue Type: Improvement Components: ipc Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: D2199.1.patch The HbaseServer queues responses to client (if the client is slow). Expose a metric that records the size of the response queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5543) Add a keepalive option for IPC connections
[
https://issues.apache.org/jira/browse/HBASE-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225566#comment-13225566
]
stack commented on HBASE-5543:
--
Yeah, it looks like its inevitable that we'll ask the server to do legitimate
stuff that will take longer than the rpctimeout yet the server is making
headway: e.g. the reproducing test case, though a little artificial, for
HBASE-4890 fix possible NPE in HConnectionManager was asking the
regionserver to open 3k regions.
If its a task like the above, there should be a facility for telling client
we're alive still or we should just refuse the request because it will take too
long (The latter we need to do t -- from Benoiit. If server is going to
take too long servicing a request, so long the client will be gone by the time
its done its work, then refuse the request... don't do the increment or update
that the updating client will not be around to see).
Add a keepalive option for IPC connections
--
Key: HBASE-5543
URL: https://issues.apache.org/jira/browse/HBASE-5543
Project: HBase
Issue Type: Improvement
Components: client, coprocessors, ipc
Reporter: Andrew Purtell
On the user list someone wrote in with a connection failure due to a long
running coprocessor:
{quote}
On Wed, Mar 7, 2012 at 10:59 PM, raghavendhra rahul wrote:
2012-03-08 12:03:09,475 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call execCoprocessor([B@50cb21, getProjection(), rpc version=1,
client version=0, methodsFingerPrint=0), rpc version=1, client version=29,
methodsFingerPrint=54742778 from 10.184.17.26:46472: output error
2012-03-08 12:03:09,476 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020 caught: java.nio.channels.ClosedChannelException
{quote}
I suggested in response we might consider give our RPC a keepalive option for
calls that may run for a long time (like execCoprocessor).
LarsH +1ed the idea:
{quote}
+1 on keepalive. It's a shame (especially for long running server code) to
do all the work, just to find out at the end that the client has given up.
Or maybe there should be a way to cancel an operation if the clients decides
it does not want to wait any longer (PostgreSQL does that for example). Here
that would mean the server would need to check periodically and coprocessors
would need to be written to support that - so maybe that's no-starter.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5529) MR test failures becuase MALLOC_ARENA_MAX is not set
[ https://issues.apache.org/jira/browse/HBASE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225574#comment-13225574 ] Hudson commented on HBASE-5529: --- Integrated in HBase-0.94 #20 (See [https://builds.apache.org/job/HBase-0.94/20/]) HBASE-5529 MR test failures becuase MALLOC_ARENA_MAX is not set (Gregory Chanan) (Revision 1298575) Result = SUCCESS larsh : Files : * /hbase/branches/0.94/pom.xml MR test failures becuase MALLOC_ARENA_MAX is not set Key: HBASE-5529 URL: https://issues.apache.org/jira/browse/HBASE-5529 Project: HBase Issue Type: Bug Components: mapreduce, test Affects Versions: 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor Fix For: 0.92.1, 0.94.0, 0.96.0 Attachments: HBASE-5529-to92.patch, HBASE-5529-trunk.patch When running unit tests on CentOS 6 I get a bunch of unit test failures in mapreduce-related tests due to: 2012-03-03 00:14:18,776 WARN [Container Monitor] monitor.ContainersMonitorImpl$MonitoringThread(436): Container [pid=21446,containerID=container_1330762435849_0002_01_01] is running beyond virtual memory limits. Current usage: 223.1mb of 2.0gb physical memory used; 6.9gb of 4.2gb virtual memory used. Killing container. Note: this also came up in the mapreduce project. See: https://issues.apache.org/jira/browse/MAPREDUCE-3933 Patch coming shortly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5509) MR based copier for copying HFiles (trunk version)
[ https://issues.apache.org/jira/browse/HBASE-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225580#comment-13225580 ] Zhihong Yu commented on HBASE-5509: --- I put a few comments on RB. MR based copier for copying HFiles (trunk version) -- Key: HBASE-5509 URL: https://issues.apache.org/jira/browse/HBASE-5509 Project: HBase Issue Type: Sub-task Components: documentation, regionserver Reporter: Karthik Ranganathan Assignee: Lars Hofhansl Fix For: 0.94.0, 0.96.0 Attachments: 5509-v2.txt, 5509.txt This copier is a modification of the distcp tool in HDFS. It does the following: 1. List out all the regions in the HBase cluster for the required table 2. Write the above out to a file 3. Each mapper 3.1 lists all the HFiles for a given region by querying the regionserver 3.2 copies all the HFiles 3.3 outputs success if the copy succeeded, failure otherwise. Failed regions are retried in another loop 4. Mappers are placed on nodes which have maximum locality for a given region to speed up copying -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5522) hbase 0.92 test artifacts are missing from Maven central
[ https://issues.apache.org/jira/browse/HBASE-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225596#comment-13225596 ] Enis Soztutar commented on HBASE-5522: -- @Roman, Are you able to fetch the test artifacts now? hbase 0.92 test artifacts are missing from Maven central Key: HBASE-5522 URL: https://issues.apache.org/jira/browse/HBASE-5522 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.92.0 Reporter: Roman Shaposhnik Assignee: stack Fix For: 0.92.1, 0.94.0 Attachments: 5522.txt Could someone with enough karma, please, publish the test artifacts for 0.92.0? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225604#comment-13225604 ] Hadoop QA commented on HBASE-5526: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517616/java_HBASE-5526-v7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -127 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.TestZooKeeper Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1138//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1138//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1138//console This message is automatically generated. Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225640#comment-13225640 ] Phabricator commented on HBASE-5074: mbautin has committed the revision [jira] [HBASE-5074] Support checksums in HBase block cache. REVISION DETAIL https://reviews.facebook.net/D1521 COMMIT https://reviews.facebook.net/rHBASE1298641 support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225654#comment-13225654 ] stack commented on HBASE-5074: -- Wahoo!! Lars, you want to pull it into 0.94? (Does this mean 0.94 is good to go? Should we put up an RC?) support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225655#comment-13225655 ] Lars Hofhansl commented on HBASE-5074: -- Yes sir. Still waiting for HBASE-4608. And HBASE-5541 :) support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225659#comment-13225659 ] Lars Hofhansl commented on HBASE-5526: -- Ok... last chance to object. Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
[ https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225660#comment-13225660 ] Lars Hofhansl commented on HBASE-5541: -- Looks good to me. Any objections to committing this? Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks Key: HBASE-5541 URL: https://issues.apache.org/jira/browse/HBASE-5541 Project: HBase Issue Type: Sub-task Components: client, regionserver Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 5541.txt Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed. Similar to what we do in doMiniBatchPut, we should create the log entry with the lock held, but only sync the HLog after the lock is released, along with rollback logic in case the sync'ing fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225658#comment-13225658 ] Lars Hofhansl commented on HBASE-5074: -- and HBASE-5526, but that's it from my side. support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
[ https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225661#comment-13225661 ] Zhihong Yu commented on HBASE-5541: --- Redundant step 11 comment would be removed, right ? Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks Key: HBASE-5541 URL: https://issues.apache.org/jira/browse/HBASE-5541 Project: HBase Issue Type: Sub-task Components: client, regionserver Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 5541.txt Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed. Similar to what we do in doMiniBatchPut, we should create the log entry with the lock held, but only sync the HLog after the lock is released, along with rollback logic in case the sync'ing fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225666#comment-13225666 ] Jesse Yates commented on HBASE-5526: Btw, all failed tests passed locally. Do we want to add any extra documentation to ref guide or do you think the standard stuff from hbase-defaults.xml is going to suffice? Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
[ https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225667#comment-13225667 ] Lars Hofhansl commented on HBASE-5541: -- Yep :) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks Key: HBASE-5541 URL: https://issues.apache.org/jira/browse/HBASE-5541 Project: HBase Issue Type: Sub-task Components: client, regionserver Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 5541.txt Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed. Similar to what we do in doMiniBatchPut, we should create the log entry with the lock held, but only sync the HLog after the lock is released, along with rollback logic in case the sync'ing fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5529) MR test failures becuase MALLOC_ARENA_MAX is not set
[ https://issues.apache.org/jira/browse/HBASE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225678#comment-13225678 ] Hudson commented on HBASE-5529: --- Integrated in HBase-0.92-security #98 (See [https://builds.apache.org/job/HBase-0.92-security/98/]) HBASE-5529 MR test failures becuase MALLOC_ARENA_MAX is not set (Gregory Chanan) (Revision 1298582) Result = FAILURE larsh : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/pom.xml MR test failures becuase MALLOC_ARENA_MAX is not set Key: HBASE-5529 URL: https://issues.apache.org/jira/browse/HBASE-5529 Project: HBase Issue Type: Bug Components: mapreduce, test Affects Versions: 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor Fix For: 0.92.1, 0.94.0, 0.96.0 Attachments: HBASE-5529-to92.patch, HBASE-5529-trunk.patch When running unit tests on CentOS 6 I get a bunch of unit test failures in mapreduce-related tests due to: 2012-03-03 00:14:18,776 WARN [Container Monitor] monitor.ContainersMonitorImpl$MonitoringThread(436): Container [pid=21446,containerID=container_1330762435849_0002_01_01] is running beyond virtual memory limits. Current usage: 223.1mb of 2.0gb physical memory used; 6.9gb of 4.2gb virtual memory used. Killing container. Note: this also came up in the mapreduce project. See: https://issues.apache.org/jira/browse/MAPREDUCE-3933 Patch coming shortly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5529) MR test failures becuase MALLOC_ARENA_MAX is not set
[ https://issues.apache.org/jira/browse/HBASE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225679#comment-13225679 ] Hudson commented on HBASE-5529: --- Integrated in HBase-0.92 #322 (See [https://builds.apache.org/job/HBase-0.92/322/]) HBASE-5529 MR test failures becuase MALLOC_ARENA_MAX is not set (Gregory Chanan) (Revision 1298582) Result = SUCCESS larsh : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/pom.xml MR test failures becuase MALLOC_ARENA_MAX is not set Key: HBASE-5529 URL: https://issues.apache.org/jira/browse/HBASE-5529 Project: HBase Issue Type: Bug Components: mapreduce, test Affects Versions: 0.92.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor Fix For: 0.92.1, 0.94.0, 0.96.0 Attachments: HBASE-5529-to92.patch, HBASE-5529-trunk.patch When running unit tests on CentOS 6 I get a bunch of unit test failures in mapreduce-related tests due to: 2012-03-03 00:14:18,776 WARN [Container Monitor] monitor.ContainersMonitorImpl$MonitoringThread(436): Container [pid=21446,containerID=container_1330762435849_0002_01_01] is running beyond virtual memory limits. Current usage: 223.1mb of 2.0gb physical memory used; 6.9gb of 4.2gb virtual memory used. Killing container. Note: this also came up in the mapreduce project. See: https://issues.apache.org/jira/browse/MAPREDUCE-3933 Patch coming shortly -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225686#comment-13225686 ] Lars Hofhansl commented on HBASE-5074: -- Is D1521.14.patch the that was applied to trunk? support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225689#comment-13225689 ] Lars Hofhansl commented on HBASE-5526: -- Can file a child bug for documentation and why this might be useful. Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5526) Configurable file and directory based umask
[ https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5526: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to 0.94 and trunk Configurable file and directory based umask --- Key: HBASE-5526 URL: https://issues.apache.org/jira/browse/HBASE-5526 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.0 Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch Currently many all the files created by the HBase user are just written using the default file permissions granted by hdfs. However, to ensure only the correct user/group views the files and directories, we need to be able to apply a configurable umask to either directories or files. This ticket covers setting permissions for files written to dfs, as opposed to things like pid and log files. The impetus for this was to allow the web-user to view the directory structure of hbase, but not to actually see any of the actual data hbase is storing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5544) Add metrics to HRegion.processRow()
Add metrics to HRegion.processRow() --- Key: HBASE-5544 URL: https://issues.apache.org/jira/browse/HBASE-5544 Project: HBase Issue Type: New Feature Reporter: Scott Chen Assignee: Scott Chen Add metrics of 1. time for waiting for the lock 2. processing time (scan time) 3. time spent while holding the lock 4. total call time 5. number of failures / calls -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5213) hbase master stop does not bring down backup masters
[ https://issues.apache.org/jira/browse/HBASE-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-5213: -- Attachment: HBASE-5213-v2-trunk.patch HBASE-5213-v2-92.patch HBASE-5213-v2-90.patch * Attached v2 for trunk,92, and 90 * For trunk, only small differences from previous version (comments and spacing). Tested that trunk patch works with 94 as well. Created patches for 92 and 90. hbase master stop does not bring down backup masters -- Key: HBASE-5213 URL: https://issues.apache.org/jira/browse/HBASE-5213 Project: HBase Issue Type: Bug Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor Attachments: HBASE-5213-v0-trunk.patch, HBASE-5213-v1-trunk.patch, HBASE-5213-v2-90.patch, HBASE-5213-v2-92.patch, HBASE-5213-v2-trunk.patch Typing hbase master stop produces the following message: stop Start cluster shutdown; Master signals RegionServer shutdown It seems like backup masters should be considered part of the cluster, but they are not brought down by hbase master stop. stop-hbase.sh does correctly bring down the backup masters. The same behavior is observed when a client app makes use of the client API HBaseAdmin.shutdown() http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#shutdown() -- this isn't too surprising since I think hbase master stop just calls this API. It seems like HBASE-1448 address this; perhaps there was a regression? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225703#comment-13225703 ] Mikhail Bautin commented on HBASE-5074: --- @Lars: yes. I have also re-run unit tests one more time. support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225704#comment-13225704 ] Lars Hofhansl commented on HBASE-5074: -- Thanks I'll apply and commit the patch to 0.94. Thanks for the great work guys!! support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5213) hbase master stop does not bring down backup masters
[ https://issues.apache.org/jira/browse/HBASE-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13225712#comment-13225712 ] Zhihong Yu commented on HBASE-5213: --- I ran TestAdmin three times based on patch v2. They passed. hbase master stop does not bring down backup masters -- Key: HBASE-5213 URL: https://issues.apache.org/jira/browse/HBASE-5213 Project: HBase Issue Type: Bug Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor Attachments: HBASE-5213-v0-trunk.patch, HBASE-5213-v1-trunk.patch, HBASE-5213-v2-90.patch, HBASE-5213-v2-92.patch, HBASE-5213-v2-trunk.patch Typing hbase master stop produces the following message: stop Start cluster shutdown; Master signals RegionServer shutdown It seems like backup masters should be considered part of the cluster, but they are not brought down by hbase master stop. stop-hbase.sh does correctly bring down the backup masters. The same behavior is observed when a client app makes use of the client API HBaseAdmin.shutdown() http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#shutdown() -- this isn't too surprising since I think hbase master stop just calls this API. It seems like HBASE-1448 address this; perhaps there was a regression? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5074: - Attachment: 5074-0.94.txt Here's the 0.94 version. Applied mostly with some offsets, just had to fix up some imports. support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: 5074-0.94.txt, D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5074: - Status: Open (was: Patch Available) support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: 5074-0.94.txt, D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
