[jira] [Commented] (HBASE-3614) Expose per-region request rate metrics
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258013#comment-13258013 ] Todd Lipcon commented on HBASE-3614: Gotcha. Apologies for my cluelessness :) Seems fine. Though I think Shaneal's percentile stuff would be a nice improvement, what we've got now is at least useful. > Expose per-region request rate metrics > -- > > Key: HBASE-3614 > URL: https://issues.apache.org/jira/browse/HBASE-3614 > Project: HBase > Issue Type: Improvement > Components: metrics, regionserver >Reporter: Gary Helmling >Assignee: Elliott Clark >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, > HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, > HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, > HBASE-3614-8.patch, HBASE-3614-9.patch, Screen Shot 2012-04-17 at 2.41.27 > PM.png > > > We currently export metrics on request rates for each region server, and this > can help with identifying uneven load at a high level. But once you see a > given server under high load, you're forced to extrapolate based on your > application patterns and the data it's serving what the likely culprit is. > This can and should be much easier if we just exported request rate metrics > per-region on each server. > Dynamically updating the metrics keys based on assigned regions may pose some > minor challenges, but this seems a very valuable diagnostic tool to have > available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3614) Expose per-region request rate metrics
[ https://issues.apache.org/jira/browse/HBASE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257978#comment-13257978 ] Todd Lipcon commented on HBASE-3614: Just getting to this after it's committed.. but: what's the time scale of "avgtime" on these metrics? Averages that are since boot are kind of useless. We should either track the sum of the time, or do a time-biased metric like Shaneal added recently in other places. > Expose per-region request rate metrics > -- > > Key: HBASE-3614 > URL: https://issues.apache.org/jira/browse/HBASE-3614 > Project: HBase > Issue Type: Improvement > Components: metrics, regionserver >Reporter: Gary Helmling >Assignee: Elliott Clark >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-3614-0.patch, HBASE-3614-1.patch, > HBASE-3614-2.patch, HBASE-3614-3.patch, HBASE-3614-4.patch, > HBASE-3614-5.patch, HBASE-3614-6.patch, HBASE-3614-7.patch, > HBASE-3614-8.patch, HBASE-3614-9.patch, Screen Shot 2012-04-17 at 2.41.27 > PM.png > > > We currently export metrics on request rates for each region server, and this > can help with identifying uneven load at a high level. But once you see a > given server under high load, you're forced to extrapolate based on your > application patterns and the data it's serving what the likely culprit is. > This can and should be much easier if we just exported request rate metrics > per-region on each server. > Dynamically updating the metrics keys based on assigned regions may pose some > minor challenges, but this seems a very valuable diagnostic tool to have > available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5831) hadoopqa builds not completing
[
https://issues.apache.org/jira/browse/HBASE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257719#comment-13257719
]
Todd Lipcon commented on HBASE-5831:
I agree, we should make it a requirement that all @Tests have a timeout and a
category. The coolest would be a script that looked at the previous Jenkins
results, and automatically generated a diff for all the tests, setting their
timeouts to 2x the previous runtime, so we don't have to manually guess a
timeout for all our hundreds of tests :)
> hadoopqa builds not completing
> --
>
> Key: HBASE-5831
> URL: https://issues.apache.org/jira/browse/HBASE-5831
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Attachments: 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt,
> 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt,
> 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt,
> 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt
>
>
> No test failures but build complains it has failed. trunk build seems to
> have the same affliction:
> {code}
> Results :
> Tests run: 909, Failures: 0, Errors: 0, Skipped: 9
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 41:19.273s
> [INFO] Finished at: Wed Apr 18 21:54:31 UTC 2012
> [INFO] Final Memory: 59M/451M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test
> (secondPartTestsExecution) on project hbase: Failure or timeout -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> -1 overall. Here are the results of testing the latest attachment
> http://issues.apache.org/jira/secure/attachment/12523250/5811+%281%29.txt
> against trunk revision .
> +1 @author. The patch does not contain any @author tags.
> +1 tests included. The patch appears to include 3 new or modified tests.
> +1 javadoc. The javadoc tool did not generate any warning messages.
> +1 javac. The applied patch does not increase the total number of javac
> compiler warnings.
> -1 findbugs. The patch appears to introduce 6 new Findbugs (version
> 1.3.9) warnings.
> +1 release audit. The applied patch does not increase the total number
> of release audit warnings.
> -1 core tests. The patch failed these unit tests:
> {code}
> Its not apparent that any particular test is not finishing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5831) hadoopqa builds not completing
[
https://issues.apache.org/jira/browse/HBASE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257626#comment-13257626
]
Todd Lipcon commented on HBASE-5831:
Maybe adding a "timeout=6" or so to the test would let it properly get
marked as failed without screwing the whole build? I've found that surefire
really sucks at handling test timeouts unless they also have junit annotations
with a timeout.
> hadoopqa builds not completing
> --
>
> Key: HBASE-5831
> URL: https://issues.apache.org/jira/browse/HBASE-5831
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Attachments: 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt,
> 5831.remove.TestLoadIncrementalHFilesSplitRecovery.txt
>
>
> No test failures but build complains it has failed. trunk build seems to
> have the same affliction:
> {code}
> Results :
> Tests run: 909, Failures: 0, Errors: 0, Skipped: 9
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 41:19.273s
> [INFO] Finished at: Wed Apr 18 21:54:31 UTC 2012
> [INFO] Final Memory: 59M/451M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test
> (secondPartTestsExecution) on project hbase: Failure or timeout -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> -1 overall. Here are the results of testing the latest attachment
> http://issues.apache.org/jira/secure/attachment/12523250/5811+%281%29.txt
> against trunk revision .
> +1 @author. The patch does not contain any @author tags.
> +1 tests included. The patch appears to include 3 new or modified tests.
> +1 javadoc. The javadoc tool did not generate any warning messages.
> +1 javac. The applied patch does not increase the total number of javac
> compiler warnings.
> -1 findbugs. The patch appears to introduce 6 new Findbugs (version
> 1.3.9) warnings.
> +1 release audit. The applied patch does not increase the total number
> of release audit warnings.
> -1 core tests. The patch failed these unit tests:
> {code}
> Its not apparent that any particular test is not finishing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5826) Improve sync of HLog edits
[ https://issues.apache.org/jira/browse/HBASE-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257159#comment-13257159 ] Todd Lipcon commented on HBASE-5826: I'm happy to work on this, if folks would like. But, I probably won't be able to commit a lot of time - so if people are antsy to get this sooner rather than later, feel free to steal it from me :) > Improve sync of HLog edits > -- > > Key: HBASE-5826 > URL: https://issues.apache.org/jira/browse/HBASE-5826 > Project: HBase > Issue Type: Improvement >Reporter: Zhihong Yu > > HBASE-5782 solved the correctness issue for the sync of HLog edits. > Todd provided a patch that would achieve higher throughput. > This JIRA is a continuation of Todd's work submitted there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid order since HBASE-4487
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256030#comment-13256030 ] Todd Lipcon commented on HBASE-5782: I think for 0.94.0 we should commit Lars's, and file another JIRA to clean up / rewrite this stuff to be less circuitous. Could definitely do with a cleanup. > Edits can be appended out of seqid order since HBASE-4487 > - > > Key: HBASE-5782 > URL: https://issues.apache.org/jira/browse/HBASE-5782 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 0.94.0 >Reporter: Gopinathan A >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.94.0 > > Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, > 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt > > > Create a table with 1000 splits, after the region assignemnt, kill the > regionserver wich contains META table. > Here few regions are missing after the log splitting and region assigment. > HBCK report shows multiple region holes are got created. > Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid order since HBASE-4487
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255832#comment-13255832 ] Todd Lipcon commented on HBASE-5782: Seems reasonable to me. Did yours pass tests, etc? > Edits can be appended out of seqid order since HBASE-4487 > - > > Key: HBASE-5782 > URL: https://issues.apache.org/jira/browse/HBASE-5782 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 0.94.0 >Reporter: Gopinathan A >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.94.0 > > Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, > 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt > > > Create a table with 1000 splits, after the region assignemnt, kill the > regionserver wich contains META table. > Here few regions are missing after the log splitting and region assigment. > HBCK report shows multiple region holes are got created. > Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid order since HBASE-4487
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255718#comment-13255718 ] Todd Lipcon commented on HBASE-5782: Don't think you're smoking something. It looks reasonable, though I think there are still races around the update of doneUpTo, syncedTilHere, etc. These races may be "safe" in that they only result in too-low values (and thus extra needless syncs) but I'm not 100% sure of it. Maybe we can extend the verification test case to run as a stress test, including rolls, and run it overnight or something? > Edits can be appended out of seqid order since HBASE-4487 > - > > Key: HBASE-5782 > URL: https://issues.apache.org/jira/browse/HBASE-5782 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 0.94.0 >Reporter: Gopinathan A >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.94.0 > > Attachments: 5782-lars-v2.txt, 5782-sketch.txt, 5782.txt, > 5782.unfinished-stack.txt, HBASE-5782.patch, hbase-5782.txt > > > Create a table with 1000 splits, after the region assignemnt, kill the > regionserver wich contains META table. > Here few regions are missing after the log splitting and region assigment. > HBCK report shows multiple region holes are got created. > Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5782) Edits can be appended out of seqid order since HBASE-4487
[ https://issues.apache.org/jira/browse/HBASE-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255395#comment-13255395 ] Todd Lipcon commented on HBASE-5782: Looks like some test timeouts... will investigate tomorrow. > Edits can be appended out of seqid order since HBASE-4487 > - > > Key: HBASE-5782 > URL: https://issues.apache.org/jira/browse/HBASE-5782 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 0.94.0 >Reporter: Gopinathan A >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.94.0 > > Attachments: 5782-sketch.txt, 5782.txt, 5782.unfinished-stack.txt, > HBASE-5782.patch, hbase-5782.txt > > > Create a table with 1000 splits, after the region assignemnt, kill the > regionserver wich contains META table. > Here few regions are missing after the log splitting and region assigment. > HBCK report shows multiple region holes are got created. > Same scenario was verified mulitple times in 0.92.1, no issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5792) HLog Performance Evaluation Tool
[ https://issues.apache.org/jira/browse/HBASE-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255337#comment-13255337 ] Todd Lipcon commented on HBASE-5792: I just noticed we already have a "TestHLogBench" tool which appears pretty close to this... we should merge the two in some way or be clear on what the difference is. > HLog Performance Evaluation Tool > > > Key: HBASE-5792 > URL: https://issues.apache.org/jira/browse/HBASE-5792 > Project: HBase > Issue Type: Test > Components: wal >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Labels: performance, wal > Fix For: 0.96.0 > > Attachments: HBASE-5792-v0.patch, HBASE-5792-v1.patch, > HBASE-5792-v2.patch, verify.txt, verify.txt > > > Related to HDFS-3280 and the HBase WAL slowdown on 0.23+ > It would be nice to have a simple tool like HFilePerformanceEvaluation, ... > to be able to check easily the HLog performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254961#comment-13254961 ] Todd Lipcon commented on HBASE-5620: Did anyone do any benchmarks here? It concerns me to have this committed without even knowing how it affects performance. > Convert the client protocol of HRegionInterface to PB > - > > Key: HBASE-5620 > URL: https://issues.apache.org/jira/browse/HBASE-5620 > Project: HBase > Issue Type: Sub-task > Components: ipc, master, migration, regionserver >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.96.0 > > Attachments: hbase-5620-sec.patch, hbase-5620_v3.patch, > hbase-5620_v4.patch, hbase-5620_v4.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5792) HLog Performance Evaluation Tool
[ https://issues.apache.org/jira/browse/HBASE-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254496#comment-13254496 ] Todd Lipcon commented on HBASE-5792: bq. I wonder if it makes sense to persist benchmark results to HBase. Let's not scope creep. The point of this tool is to have a quick benchmark you can run without having to set up an HBase cluster. Outputting results on the console is good enough to parse and store in whatever database you want. My preference would probably be just appending to a text file for easy graphing from Hudson. > HLog Performance Evaluation Tool > > > Key: HBASE-5792 > URL: https://issues.apache.org/jira/browse/HBASE-5792 > Project: HBase > Issue Type: Test > Components: wal >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Labels: performance, wal > Attachments: HBASE-5792-v0.patch, HBASE-5792-v1.patch > > > Related to HDFS-3280 and the HBase WAL slowdown on 0.23+ > It would be nice to have a simple tool like HFilePerformanceEvaluation, ... > to be able to check easily the HLog performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5786) Implement histogram metrics for flush and compaction latencies and sizes.
[ https://issues.apache.org/jira/browse/HBASE-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253831#comment-13253831 ] Todd Lipcon commented on HBASE-5786: Or even keep a round robin buffer of the last 1000 such operations? It would be a miniscule amount of RAM to track this, right? > Implement histogram metrics for flush and compaction latencies and sizes. > - > > Key: HBASE-5786 > URL: https://issues.apache.org/jira/browse/HBASE-5786 > Project: HBase > Issue Type: New Feature > Components: metrics, regionserver >Affects Versions: 0.92.2, 0.94.0, 0.96.0 >Reporter: Jonathan Hsieh >Assignee: Shaneal Manek > > Average time for region operations doesn't really tell a useful story when > that help diagnose anomalous conditions. > It would be extremely useful to add histogramming metrics similar to > HBASE-5533 for region operations like flush, compaction and splitting. The > probably should be forward biased at a much coarser granularity however > (maybe decay every day?) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253829#comment-13253829 ] Todd Lipcon commented on HBASE-5776: Gotcha. But is this a user-facing API? or would the HTables themselves write into an HTable multiplexer? It seems to me like this kind of behavior should happen automatically for any writes going into HBase, without expanding our API footprint. Is there some reason that that's impossible? > HTableMultiplexer > -- > > Key: HBASE-5776 > URL: https://issues.apache.org/jira/browse/HBASE-5776 > Project: HBase > Issue Type: Improvement >Reporter: Liyin Tang >Assignee: Liyin Tang > Attachments: D2775.1.patch, D2775.1.patch, D2775.2.patch, > D2775.2.patch > > > There is a known issue in HBase client that single slow/dead region server > could slow down the multiput operations across all the region servers. So the > HBase client will be as slow as the slowest region server in the cluster. > > To solve this problem, HTableMultiplexer will separate the multiput > submitting threads with the flush threads, which means the multiput operation > will be a nonblocking operation. > The submitting thread will shard all the puts into different queues based on > its destination region server and return immediately. The flush threads will > flush these puts from each queue to its destination region server. > Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253122#comment-13253122 ] Todd Lipcon commented on HBASE-5776: Can this be done without adding a new interface? eg can we make the existing queueing/flushing behavior of HTable use a threadpool? > HTableMultiplexer > -- > > Key: HBASE-5776 > URL: https://issues.apache.org/jira/browse/HBASE-5776 > Project: HBase > Issue Type: Improvement >Reporter: Liyin Tang >Assignee: Liyin Tang > Attachments: D2775.1.patch, D2775.1.patch > > > There is a known issue in HBase client that single slow/dead region server > could slow down the multiput operations across all the region servers. So the > HBase client will be as slow as the slowest region server in the cluster. > > To solve this problem, HTableMultiplexer will separate the multiput > submitting threads with the flush threads, which means the multiput operation > will be a nonblocking operation. > The submitting thread will shard all the puts into different queues based on > its destination region server and return immediately. The flush threads will > flush these puts from each queue to its destination region server. > Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5778) Turn on WAL compression by default
[ https://issues.apache.org/jira/browse/HBASE-5778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252924#comment-13252924 ] Todd Lipcon commented on HBASE-5778: Do we have this in hbase-default.xml as well? if not, +1 > Turn on WAL compression by default > -- > > Key: HBASE-5778 > URL: https://issues.apache.org/jira/browse/HBASE-5778 > Project: HBase > Issue Type: Improvement >Reporter: Jean-Daniel Cryans >Priority: Blocker > Fix For: 0.94.0, 0.96.0 > > Attachments: HBASE-5778.patch > > > I ran some tests to verify if WAL compression should be turned on by default. > For a use case where it's not very useful (values two order of magnitude > bigger than the keys), the insert time wasn't different and the CPU usage 15% > higher (150% CPU usage VS 130% when not compressing the WAL). > When values are smaller than the keys, I saw a 38% improvement for the insert > run time and CPU usage was 33% higher (600% CPU usage VS 450%). I'm not sure > WAL compression accounts for all the additional CPU usage, it might just be > that we're able to insert faster and we spend more time in the MemStore per > second (because our MemStores are bad when they contain tens of thousands of > values). > Those are two extremes, but it shows that for the price of some CPU we can > save a lot. My machines have 2 quads with HT, so I still had a lot of idle > CPUs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index
[ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249927#comment-13249927 ] Todd Lipcon commented on HBASE-5723: This seems to give even fewer guarantees than HBASE-3340 (which as I understood it used the WAL to ensure reasonably timely consistency of the index table). I'm still of the opinion that these things should start as separate projects, rather than inside HBase core, and only if we see a high demand for them and good maintenance, we should pull it in. I don't think there is much demand for non-consistent indexes. > Simple Design of Secondary Index > > > Key: HBASE-5723 > URL: https://issues.apache.org/jira/browse/HBASE-5723 > Project: HBase > Issue Type: New Feature > Components: coprocessors >Reporter: ShiXing >Priority: Minor > Attachments: Simple Design of HBase SecondaryIndex.pdf > > > Use coprocessor to create index. And primary tables' compaction to purge the > stale data. > Attach file is the Design of the Seconday Index. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5743) Support GIT patches
[ https://issues.apache.org/jira/browse/HBASE-5743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248935#comment-13248935 ] Todd Lipcon commented on HBASE-5743: +1, looks good to me. Make sure you get the +x flag for the new script on commit. > Support GIT patches > --- > > Key: HBASE-5743 > URL: https://issues.apache.org/jira/browse/HBASE-5743 > Project: HBase > Issue Type: Bug >Reporter: Nicolas Spiegelberg >Assignee: Nicolas Spiegelberg > Fix For: 0.94.0, 0.96.0, 0.89-fb > > Attachments: HBASE-5743-2.patch, HBASE-5743.patch > > > Need to import HADOOP-7384 into our version of Hadoop QA to allow test-patch > to be more flexible about patch format -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5738) Using HBase with HA HDFS requires bogus hardcoded port value
[
https://issues.apache.org/jira/browse/HBASE-5738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248083#comment-13248083
]
Todd Lipcon commented on HBASE-5738:
bq. The problem is that the HDFS client changed the fs.default.name property to
fs.defaultFS in HA.
That's not quite right -- the config key change was from 0.20 to 0.21, but the
old config key should still work. So I'm confused why this fixes the problem...
Also, this might break 0.20 (if it doesnt, then I'm suspicious of why this code
is there at all!)
> Using HBase with HA HDFS requires bogus hardcoded port value
>
>
> Key: HBASE-5738
> URL: https://issues.apache.org/jira/browse/HBASE-5738
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.92.1, 0.94.0
>Reporter: Shaneal Manek
>Assignee: Shaneal Manek
> Attachments: HBASE-5738.patch
>
>
> When configuring HBase with HDFS HA, we currently have to have the 8020 port
> (regardless of what port HDFS is using for the namenode rpc address) in the
> following property in hbase-site.xml:
> {noformat}
>
> hbase.rootdir
> hdfs://ha-nn-uri:8020/hbase
>
> {noformat}
> Otherwise the master and regionservers will not start.
> The value in the above property should really just be
> "hdfs://ha-nn-uri/hbase" (replace "ha-nn-uri" with your uri and "hbase" with
> the name of the hbase directory in HDFS that you are using, as appropriate).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index
[ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248024#comment-13248024 ] Todd Lipcon commented on HBASE-5723: If this implementation isn't giving you any consistency, then why is it any better than just having the client do two puts? I think for us to consider adding an indexing library to HBase proper, it needs to have stronger semantics. The idea that an index may be incorrect in the case of failures does not seem like something we should ship. > Simple Design of Secondary Index > > > Key: HBASE-5723 > URL: https://issues.apache.org/jira/browse/HBASE-5723 > Project: HBase > Issue Type: New Feature > Components: coprocessors >Reporter: ShiXing >Priority: Minor > Attachments: Simple Design of HBase SecondaryIndex.pdf > > > Use coprocessor to create index. And primary tables' compaction to purge the > stale data. > Attach file is the Design of the Seconday Index. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index
[ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247438#comment-13247438 ] Todd Lipcon commented on HBASE-5723: The design doc doesn't make any reference to consistency. How do you ensure consistency of the index? To do proper indexes you really need cross-table transactions, or at least a mechanism for eventual consistency if that's all you plan on providing. > Simple Design of Secondary Index > > > Key: HBASE-5723 > URL: https://issues.apache.org/jira/browse/HBASE-5723 > Project: HBase > Issue Type: New Feature > Components: coprocessors >Reporter: ShiXing >Priority: Minor > Attachments: Simple Design of HBase SecondaryIndex.pdf > > > Use coprocessor to create index. And primary tables' compaction to purge the > stale data. > Attach file is the Design of the Seconday Index. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5724) Row cache of KeyValue should be cleared in readFields().
[ https://issues.apache.org/jira/browse/HBASE-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247422#comment-13247422 ] Todd Lipcon commented on HBASE-5724: Does this need to go back to 0.90.x as well? > Row cache of KeyValue should be cleared in readFields(). > > > Key: HBASE-5724 > URL: https://issues.apache.org/jira/browse/HBASE-5724 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.1 >Reporter: Teruyoshi Zenmyo >Assignee: Teruyoshi Zenmyo > Fix For: 0.94.0, 0.96.0 > > Attachments: HBASE-5724.txt > > > KeyValue does not clear its row cache in reading new values (readFields()). > Therefore, If a KeyValue (kv) which caches its row bytes reads another > KeyValue instance, kv.getRow() returns a wrong value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5716) Make HBASE-4608 easier to use
[
https://issues.apache.org/jira/browse/HBASE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246733#comment-13246733
]
Todd Lipcon commented on HBASE-5716:
bq. The one I described.
What is the measurable impact of having compression on for non-compressible
data? My guess is it's close to 0 (slight hit for trying to match non-repeating
columns into a hash, slight improvement by getting some compression on region
ids and row keys)
> Make HBASE-4608 easier to use
> -
>
> Key: HBASE-5716
> URL: https://issues.apache.org/jira/browse/HBASE-5716
> Project: HBase
> Issue Type: Improvement
>Reporter: Jean-Daniel Cryans
>Assignee: Li Pi
> Fix For: 0.96.0, 0.94.1
>
>
> HBASE-4608 is a nice feature but after playing with it for a while I think
> the following should be fixed to make it easier to use by someone who's not a
> dev:
> - Add some signal that says that the feature is turned on. Right now you can
> {{jstack | grep KeyValueCompression}} a couple of times and if you get a hit
> you definitely know it's on, but otherwise the random user wouldn't know
> without going through the jira.
> - Add documentation in the reference guide. At the minimum add
> {{hbase.regionserver.wal.enablecompression}} in there with a small
> description. Better would be to add a section in {{Appendix B}} or something
> like that and describe the functionality a bit and who it's useful for. For
> example, flush from your brain the knowledge of the patch and read the name
> of the configuration... now let's say you have a use case that involves
> writing easily compressible values. Any normal user would believe that this
> is a good tuning parameter for them, but it's just going to waste CPU cycles.
> - Add some metrics like we have for HFiles where you get a clue about the
> compression ratio.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5716) Make HBASE-4608 easier to use
[
https://issues.apache.org/jira/browse/HBASE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246601#comment-13246601
]
Todd Lipcon commented on HBASE-5716:
My guess is that, if we trust the feature, we should turn it on by default. Is
there any situation in which using it isn't a good idea?
> Make HBASE-4608 easier to use
> -
>
> Key: HBASE-5716
> URL: https://issues.apache.org/jira/browse/HBASE-5716
> Project: HBase
> Issue Type: Improvement
>Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0, 0.94.1
>
>
> HBASE-4608 is a nice feature but after playing with it for a while I think
> the following should be fixed to make it easier to use by someone who's not a
> dev:
> - Add some signal that says that the feature is turned on. Right now you can
> {{jstack | grep KeyValueCompression}} a couple of times and if you get a hit
> you definitely know it's on, but otherwise the random user wouldn't know
> without going through the jira.
> - Add documentation in the reference guide. At the minimum add
> {{hbase.regionserver.wal.enablecompression}} in there with a small
> description. Better would be to add a section in {{Appendix B}} or something
> like that and describe the functionality a bit and who it's useful for. For
> example, flush from your brain the knowledge of the patch and read the name
> of the configuration... now let's say you have a use case that involves
> writing easily compressible values. Any normal user would believe that this
> is a good tuning parameter for them, but it's just going to waste CPU cycles.
> - Add some metrics like we have for HFiles where you get a clue about the
> compression ratio.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4254) Get tests passing on Hadoop 23
[ https://issues.apache.org/jira/browse/HBASE-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244856#comment-13244856 ] Todd Lipcon commented on HBASE-4254: I haven't had time to really dig into this yet -- lots of details to sift through. But I think the idea of calling recoverLease anywhere we read from a WAL isn't quite right, since it will probably break replication. We read from open WALs in that circumstance, and calling recoverLease under an open file will kill the writer. > Get tests passing on Hadoop 23 > -- > > Key: HBASE-4254 > URL: https://issues.apache.org/jira/browse/HBASE-4254 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.92.2 > > Attachments: HBASE-4254-92.patch > > > Currently some 30 or so tests are failing on the "HBase-trunk-on-hadoop-23" > build. It looks like most are reflection-based issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244489#comment-13244489 ] Todd Lipcon commented on HBASE-3909: I think adding a "refreshConfigs" admin command is a good idea. It can re-read the configs off the local disk, and emit warnings for any configs that changed that were not runtime-reconfigurable. > Add dynamic config > -- > > Key: HBASE-3909 > URL: https://issues.apache.org/jira/browse/HBASE-3909 > Project: HBase > Issue Type: Bug >Reporter: stack > Fix For: 0.96.0 > > > I'm sure this issue exists already, at least as part of the discussion around > making online schema edits possible, but no hard this having its own issue. > Ted started a conversation on this topic up on dev and Todd suggested we > lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5688) Convert zk root-region-server znode content to pb
[ https://issues.apache.org/jira/browse/HBASE-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242957#comment-13242957 ] Todd Lipcon commented on HBASE-5688: less likely to be corrupt. This is a common technique: http://en.wikipedia.org/wiki/Magic_number_(programming)#Magic_numbers_in_files > Convert zk root-region-server znode content to pb > - > > Key: HBASE-5688 > URL: https://issues.apache.org/jira/browse/HBASE-5688 > Project: HBase > Issue Type: Task >Reporter: stack > Attachments: 5688.txt > > > Move the root-region-server znode content from the versioned bytes that > ServerName.getVersionedBytes outputs to instead be pb. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5688) Convert zk root-region-server znode content to pb
[ https://issues.apache.org/jira/browse/HBASE-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242946#comment-13242946 ] Todd Lipcon commented on HBASE-5688: If you catch the exception, you can't distinguish between a legitimate error in the data (somehow got corrupted) vs just talking to an earlier cluster which is not protobuf-enabled. Same goes for future compatibility in the case that we ever move away from protobuf to something else. > Convert zk root-region-server znode content to pb > - > > Key: HBASE-5688 > URL: https://issues.apache.org/jira/browse/HBASE-5688 > Project: HBase > Issue Type: Task >Reporter: stack > Attachments: 5688.txt > > > Move the root-region-server znode content from the versioned bytes that > ServerName.getVersionedBytes outputs to instead be pb. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5688) Convert zk root-region-server znode content to pb
[
https://issues.apache.org/jira/browse/HBASE-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242929#comment-13242929
]
Todd Lipcon commented on HBASE-5688:
bq. Tell me more about your PBUF idea. You mean a first field in each message
that is always PBUF
Something like:
{code}
public byte[] protoToZKData(Message m) {
return Bytes.concat(Bytes.toString("PBUF"), m.toByteArray())
}
{code}
i.e literally prepend the bytes to the encoded protobuf.
Then when we read, ensure that the first four bytes are PBUF. If not, we can
barf and say "looks like this cluster is from the wrong version of HBase, it's
missing the PBUF magic header on ZK node /hbase/root-server". Without the magic
number, we'd end up getting a weird decoding exception
> Convert zk root-region-server znode content to pb
> -
>
> Key: HBASE-5688
> URL: https://issues.apache.org/jira/browse/HBASE-5688
> Project: HBase
> Issue Type: Task
>Reporter: stack
> Attachments: 5688.txt
>
>
> Move the root-region-server znode content from the versioned bytes that
> ServerName.getVersionedBytes outputs to instead be pb.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5688) Convert zk root-region-server znode content to pb
[ https://issues.apache.org/jira/browse/HBASE-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242875#comment-13242875 ] Todd Lipcon commented on HBASE-5688: Hey Stack. Looks pretty good. Quick comments: - might as well start with field id 1 instead of 2 in the protobuf - I think it's better not to move code in this same patch -- might be good cleanup but a little harder to review. The move makes the change look bigger than it really is. - for the data in ZK, we might want to consider prefixing all of the entries with a static "PBUF" byte string or something. My thinking is that, while we assume protobufs will last us forever, it will provide a back-compat path if we ever change our mind down the road. > Convert zk root-region-server znode content to pb > - > > Key: HBASE-5688 > URL: https://issues.apache.org/jira/browse/HBASE-5688 > Project: HBase > Issue Type: Task >Reporter: stack > Attachments: 5688.txt > > > Move the root-region-server znode content from the versioned bytes that > ServerName.getVersionedBytes outputs to instead be pb. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240632#comment-13240632 ] Todd Lipcon commented on HBASE-3996: bq. hbase jar on job tracker is updated to include the versioning mechanism but the job client has pre-versioning hbase jar. The jar on the JT doesn't matter. Split computation and interpretation happens only in the user code -- i.e on the client machine and inside the tasks themselves. So you don't need HBase installed on the JT at all. As for the TTs, it's possible to configure the TTs to put an hbase jar on the classpath, but I usually recommend against it for the exact reason you're mentioning - if the jars differ in version, and they're not 100% API compatible, you can get nasty errors. The recommended deployment is to _not_ put hbase on the TT classpath, and instead ship the HBase dependencies as part of the MR job, using the provided utility function in TableMapReduceUtil. > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner >Assignee: Eran Kutner > Fix For: 0.96.0 > > Attachments: 3996-v2.txt, 3996-v3.txt, 3996-v4.txt, 3996-v5.txt, > 3996-v6.txt, 3996-v7.txt, HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240599#comment-13240599 ] Todd Lipcon commented on HBASE-3996: The other question is whether we need version compatibility at all for this enum. The split object is created when you submit the job, and then only used by that one job, right? i.e it's never persisted or transferred over the wire to some other process, is it? > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner >Assignee: Eran Kutner > Fix For: 0.96.0 > > Attachments: 3996-v2.txt, 3996-v3.txt, 3996-v4.txt, 3996-v5.txt, > 3996-v6.txt, 3996-v7.txt, HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240592#comment-13240592 ] Todd Lipcon commented on HBASE-3996: Rather than do manual versioning, why not switch this to a protobuf? Then you avoid the manual serialization and you don't have to worry about versioning. > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner >Assignee: Eran Kutner > Fix For: 0.96.0 > > Attachments: 3996-v2.txt, 3996-v3.txt, 3996-v4.txt, 3996-v5.txt, > 3996-v6.txt, 3996-v7.txt, HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5602) Add cache access pattern statistics and report hot blocks/keys
[ https://issues.apache.org/jira/browse/HBASE-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233101#comment-13233101 ] Todd Lipcon commented on HBASE-5602: An alternative cache metric that would be interesting to know is the last access time of the entries being evicted. If the last-access-time is only a minute or so, it's likely that you can benefit from adding more cache. If it's substantially older, then adding cache won't really help. http://en.wikipedia.org/wiki/Five-minute_rule > Add cache access pattern statistics and report hot blocks/keys > -- > > Key: HBASE-5602 > URL: https://issues.apache.org/jira/browse/HBASE-5602 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin > > In many practical applications it would be very useful to know how well > utilized the block cache is, i.e. how many times we actually access a block > once it gets into cache. This would also allow to evaluate cache-on-write on > flush. In addition, we need to keep track of and report some set of hottest > block in cache, and possibly even hottest keys. This would allow to diagnose > "hot-row" problems in real time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4608) HLog Compression
[ https://issues.apache.org/jira/browse/HBASE-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229051#comment-13229051 ] Todd Lipcon commented on HBASE-4608: btw, +1 on this new patch after you've double-checked with your logs and run it through the full suite. Lars, did you want to take a look tomorrow before it's committed? > HLog Compression > > > Key: HBASE-4608 > URL: https://issues.apache.org/jira/browse/HBASE-4608 > Project: HBase > Issue Type: New Feature >Reporter: Li Pi >Assignee: stack > Fix For: 0.94.0 > > Attachments: 4608-v19.txt, 4608-v20.txt, 4608-v22.txt, 4608v1.txt, > 4608v13.txt, 4608v13.txt, 4608v14.txt, 4608v15.txt, 4608v16.txt, 4608v17.txt, > 4608v18.txt, 4608v23.txt, 4608v24.txt, 4608v25.txt, 4608v27.txt, 4608v5.txt, > 4608v6.txt, 4608v7.txt, 4608v8fixed.txt, hbase-4608-v28-delta.txt, > hbase-4608-v28.txt > > > The current bottleneck to HBase write speed is replicating the WAL appends > across different datanodes. We can speed up this process by compressing the > HLog. Current plan involves using a dictionary to compress table name, region > id, cf name, and possibly other bits of repeated data. Also, HLog format may > be changed in other ways to produce a smaller HLog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4608) HLog Compression
[ https://issues.apache.org/jira/browse/HBASE-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228108#comment-13228108 ] Todd Lipcon commented on HBASE-4608: bq. First, compression ratio is not good - at least for the data written by PE. I saw ~40% compression on a YCSB load. So some workloads may have good results whereas others didn't. Did you also re-run the test after fixing the bug? Maybe that skewed the results? bq. Second, HLogKey persistence becomes dependent on the compression implementation. This would make plugging other compression techniques hard. I agree we should use a metadata field in the log to describe which compression mechanism is being used. > HLog Compression > > > Key: HBASE-4608 > URL: https://issues.apache.org/jira/browse/HBASE-4608 > Project: HBase > Issue Type: New Feature >Reporter: Li Pi >Assignee: Li Pi > Fix For: 0.94.0 > > Attachments: 4608-v19.txt, 4608-v20.txt, 4608-v22.txt, 4608v1.txt, > 4608v13.txt, 4608v13.txt, 4608v14.txt, 4608v15.txt, 4608v16.txt, 4608v17.txt, > 4608v18.txt, 4608v5.txt, 4608v6.txt, 4608v7.txt, 4608v8fixed.txt > > > The current bottleneck to HBase write speed is replicating the WAL appends > across different datanodes. We can speed up this process by compressing the > HLog. Current plan involves using a dictionary to compress table name, region > id, cf name, and possibly other bits of repeated data. Also, HLog format may > be changed in other ways to produce a smaller HLog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5564) Bulkload is discarding duplicate records
[ https://issues.apache.org/jira/browse/HBASE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227742#comment-13227742 ] Todd Lipcon commented on HBASE-5564: I think it's a feature, not a bug, that the timestamps are all identical. The whole point is that, in a bulk-load-only workflow, you can identify each bulk load exactly, and correlate it to the MR job that inserted it. If you want to use custom timestamps, you should specify a timestamp column in your data, or write your own MR job (ImportTsv is just an example which use useful for some cases, but for anything advanced I would expect users to write their own code) > Bulkload is discarding duplicate records > > > Key: HBASE-5564 > URL: https://issues.apache.org/jira/browse/HBASE-5564 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0 > Environment: HBase 0.92 >Reporter: Laxman >Assignee: Laxman > Labels: bulkloader > > Duplicate records are getting discarded when duplicate records exists in same > input file and more specifically if they exists in same split. > Duplicate records are considered if the records are from diffrent different > splits. > Version under test: HBase 0.92 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4818) HBase Shell - Add support for formatting row keys before output
[
https://issues.apache.org/jira/browse/HBASE-4818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226311#comment-13226311
]
Todd Lipcon commented on HBASE-4818:
I think this should be a table property, and refer to a Java class name, rather
than doing it in ruby. Doing it in ruby only helps with shell, but doing it in
Java means we can also use it in the UIs, etc. ACCUMULO-303 is helpful
reference material.
> HBase Shell - Add support for formatting row keys before output
> ---
>
> Key: HBASE-4818
> URL: https://issues.apache.org/jira/browse/HBASE-4818
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Eran Kampf
>Priority: Trivial
> Attachments: hbase-4818.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> As many HBase users use binary row keys rather than strings to optimize
> memory consumption displaying an escaped string in the HBase shell isn't
> useful (and takes a lot of screen space)
> Allowing user to provide a row key formatter as part of the scan\get commands
> would allow developers to display the row key in a way thats makes sense for
> them.
> Example:
> scan 'stats', { ROWFORMATTER => MyRowFormatter.new }
> The row formatter simply gets the bytes array key and formats it to a string.
> Its an easy change tomake with simple monkey-patching of the shell commands
> but I would be happy to see it as part of the shell itself.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5552) Clean up our jmx view; its a bit of a mess
[ https://issues.apache.org/jira/browse/HBASE-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226308#comment-13226308 ] Todd Lipcon commented on HBASE-5552: Oh, does this not affect the other metrics published at /jmx in the servlet? Sorry for confusion. > Clean up our jmx view; its a bit of a mess > -- > > Key: HBASE-5552 > URL: https://issues.apache.org/jira/browse/HBASE-5552 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 0.92.1, 0.94.0 > > Attachments: 0.92.0jmx.png, 5552.txt, currentjmxview.png, > patchedjmxview.png > > > Fix before we release 0.92.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5552) Clean up our jmx view; its a bit of a mess
[ https://issues.apache.org/jira/browse/HBASE-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226249#comment-13226249 ] Todd Lipcon commented on HBASE-5552: Changing the nesting of beans in 0.92 and 0.94 seems like an incompatible change. I'm pretty sure it will screw up our monitoring upon upgrade (Benoit isn't the only one who uses JMX!) Is it possible to have a conf to switch this? Otherwise, should defer to 96. > Clean up our jmx view; its a bit of a mess > -- > > Key: HBASE-5552 > URL: https://issues.apache.org/jira/browse/HBASE-5552 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 0.92.1, 0.94.0 > > Attachments: 0.92.0jmx.png, 5552.txt, currentjmxview.png, > patchedjmxview.png > > > Fix before we release 0.92.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5543) Add a keepalive option for IPC connections
[
https://issues.apache.org/jira/browse/HBASE-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225573#comment-13225573
]
Todd Lipcon commented on HBASE-5543:
We have "ipc.client.ping", right? What we want is sort of the opposite
direction of that?
> Add a keepalive option for IPC connections
> --
>
> Key: HBASE-5543
> URL: https://issues.apache.org/jira/browse/HBASE-5543
> Project: HBase
> Issue Type: Improvement
> Components: client, coprocessors, ipc
>Reporter: Andrew Purtell
>
> On the user list someone wrote in with a connection failure due to a long
> running coprocessor:
> {quote}
> On Wed, Mar 7, 2012 at 10:59 PM, raghavendhra rahul wrote:
> 2012-03-08 12:03:09,475 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
> Responder, call execCoprocessor([B@50cb21, getProjection(), rpc version=1,
> client version=0, methodsFingerPrint=0), rpc version=1, client version=29,
> methodsFingerPrint=54742778 from 10.184.17.26:46472: output error
> 2012-03-08 12:03:09,476 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 60020 caught: java.nio.channels.ClosedChannelException
> {quote}
> I suggested in response we might consider give our RPC a keepalive option for
> calls that may run for a long time (like execCoprocessor).
> LarsH +1ed the idea:
> {quote}
> +1 on "keepalive". It's a shame (especially for long running server code) to
> do all the work, just to find out at the end that the client has given up.
> Or maybe there should be a way to cancel an operation if the clients decides
> it does not want to wait any longer (PostgreSQL does that for example). Here
> that would mean the server would need to check periodically and coprocessors
> would need to be written to support that - so maybe that's no-starter.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4608) HLog Compression
[ https://issues.apache.org/jira/browse/HBASE-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224808#comment-13224808 ] Todd Lipcon commented on HBASE-4608: Good test case - maybe it can be turned into a functional test in the code? > HLog Compression > > > Key: HBASE-4608 > URL: https://issues.apache.org/jira/browse/HBASE-4608 > Project: HBase > Issue Type: New Feature >Reporter: Li Pi >Assignee: Li Pi > Fix For: 0.94.0 > > Attachments: 4608-v19.txt, 4608v1.txt, 4608v13.txt, 4608v13.txt, > 4608v14.txt, 4608v15.txt, 4608v16.txt, 4608v17.txt, 4608v18.txt, 4608v5.txt, > 4608v6.txt, 4608v7.txt, 4608v8fixed.txt > > > The current bottleneck to HBase write speed is replicating the WAL appends > across different datanodes. We can speed up this process by compressing the > HLog. Current plan involves using a dictionary to compress table name, region > id, cf name, and possibly other bits of repeated data. Also, HLog format may > be changed in other ways to produce a smaller HLog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221127#comment-13221127 ] Todd Lipcon commented on HBASE-5451: Sounds reasonable, thanks Devaraj. > Switch RPC call envelope/headers to PBs > --- > > Key: HBASE-5451 > URL: https://issues.apache.org/jira/browse/HBASE-5451 > Project: HBase > Issue Type: Sub-task > Components: ipc, master, migration, regionserver >Affects Versions: 0.94.0 >Reporter: Todd Lipcon >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221124#comment-13221124 ] Todd Lipcon commented on HBASE-5074: There's no benefit to CRC32C over CRC32 unless you can use the JNI code. I don't think copy-pasting all of the JNI stuff into HBase is a good idea. And, besides, this patch is not yet equipped to do the JNI-based checksumming (which requires direct buffers, etc) > support checksums in HBase block cache > -- > > Key: HBASE-5074 > URL: https://issues.apache.org/jira/browse/HBASE-5074 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, > D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, > D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, > D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, > D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, > D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, > D1521.9.patch > > > The current implementation of HDFS stores the data in one block file and the > metadata(checksum) in another block file. This means that every read into the > HBase block cache actually consumes two disk iops, one to the datafile and > one to the checksum file. This is a major problem for scaling HBase, because > HBase is usually bottlenecked on the number of random disk iops that the > storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5494) Introduce a zk hosted table-wide read/write lock so only one table operation at a time
[ https://issues.apache.org/jira/browse/HBASE-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220591#comment-13220591 ] Todd Lipcon commented on HBASE-5494: Might be worth considering region-specific locks as well? Or is that too much scope creep? (of course we'd need to ensure a locking order to avoid deadlock) > Introduce a zk hosted table-wide read/write lock so only one table operation > at a time > -- > > Key: HBASE-5494 > URL: https://issues.apache.org/jira/browse/HBASE-5494 > Project: HBase > Issue Type: Improvement >Reporter: stack > > I saw this facility over in the accumulo code base. > Currently we just try to sort out the mess when splits come in during an > online schema edit; somehow we figure we can figure all possible region > transition combinations and make the right call. > We could try and narrow the number of combinations by taking out a zk table > lock when doing table operations. > For example, on split or merge, we could take a read-only lock meaning the > table can't be disabled while these are running. > We could then take a write only lock if we want to ensure the table doesn't > change while disabling or enabling process is happening. > Shouldn't be too hard to add. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220539#comment-13220539 ] Todd Lipcon commented on HBASE-5498: Nice idea, seems reasonable to me. > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement >Reporter: Francis Liu > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4348) Add metrics for regions in transition
[ https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220416#comment-13220416 ] Todd Lipcon commented on HBASE-4348: This patch should actually add _metrics_, not just UI entries. Adding to the UI is nice sugar, but exposing via JMX and the hadoop metrics interface is higher priority IMO. > Add metrics for regions in transition > - > > Key: HBASE-4348 > URL: https://issues.apache.org/jira/browse/HBASE-4348 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: Himanshu Vashishtha >Priority: Minor > Labels: noob > Attachments: 4348-v1.patch, 4348-v2.patch, RITs.png > > > The following metrics would be useful for monitoring the master: > - the number of regions in transition > - the number of regions in transition that have been in transition for more > than a minute > - how many seconds has the oldest region-in-transition been in transition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220250#comment-13220250 ] Todd Lipcon commented on HBASE-5451: Can we avoid the copy in the interim by having a convention that, if the request is a protobuf, then we send it _following_ the call envelope rather than inside it? (does that make sense?) > Switch RPC call envelope/headers to PBs > --- > > Key: HBASE-5451 > URL: https://issues.apache.org/jira/browse/HBASE-5451 > Project: HBase > Issue Type: Sub-task > Components: ipc, master, migration, regionserver >Affects Versions: 0.94.0 >Reporter: Todd Lipcon >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4991) Provide capability to delete named region
[ https://issues.apache.org/jira/browse/HBASE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220220#comment-13220220 ] Todd Lipcon commented on HBASE-4991: How is this a performance-critical path? Most of the use cases are around things like aging off old data, which isn't perf-critical at all. We should aim for simplicity rather than performance when things aren't in a hot path or in a time-to-recovery path. > Provide capability to delete named region > - > > Key: HBASE-4991 > URL: https://issues.apache.org/jira/browse/HBASE-4991 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Mubarak Seyed > Fix For: 0.94.0 > > Attachments: HBASE-4991.trunk.v1.patch, HBASE-4991.trunk.v2.patch > > > See discussion titled 'Able to control routing to Solr shards or not' on > lily-discuss > User may want to quickly dispose of out of date records by deleting specific > regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220186#comment-13220186 ] Todd Lipcon commented on HBASE-5498: Cool, I'll keep watching this JIRA. Thanks for working on the problem - the current workarounds are definitely annoying and less than optimal - will be good to have a true solution in place. > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement >Reporter: Francis Liu > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219915#comment-13219915 ] Todd Lipcon commented on HBASE-5498: Two things: 1) HBase currently has no dependencies on MR. One can submit MR jobs that write to/from HBase, but really those are just using the MR client from within a MR context. 2) A large proportion of bulk load use cases generate HFiles at the end of a pipeline which involves custom user code -- for example the map phase parses and processes a custom format, emitting keys into whatever structure is needed in HBase. Then, the reducer performs the partition/sort to write out the appropriate HFiles. Thus the jobs themselves are running user code, not just a pre-baked example like ImportTSV. The proposed solution doesn't address this use case -- it's totally unacceptable to run user code under an HBase security context in a multi-tenant environment. > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement >Reporter: Francis Liu > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219897#comment-13219897 ] Todd Lipcon commented on HBASE-5498: Another workaround is to build a simple service which runs with credentials that are part of the HDFS supergroup. This service would have a single API call which would chgrp a directory to an hbase group, allowing hbase to move the HFiles out of it. A third workaround would be to have HBase itself run with superuser privileges, so it could chown the user's files to hbase ownership before moving. I think all of these are vastly superior to the option where HBase itself is submitting MR jobs to perform the import. > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement >Reporter: Francis Liu > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219848#comment-13219848 ] Todd Lipcon commented on HBASE-5498: Adding a coprocessor hook to bulk load makes sense, but that seems like a generic coprocessor enhancement. The specific case of secure bulk load is a separate problem of ownership transfer, and actually is present even without the authorization enhancements, so long as HDFS permissions are enabled. Currently, our suggested workaround is that the user who generates the HFiles should chmod the output directory to be world writable, thus allowing the hbase user to move the files from the output directory into the hbase directories. This seems sufficient as a stop-gap until we can add an "ownership transfer" functionality in HDFS, no? > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement >Reporter: Francis Liu > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[
https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219708#comment-13219708
]
Todd Lipcon commented on HBASE-5498:
This seems backwards - not all bulk loads are text import jobs, so you don't
want HBase to be running the MR job. Instead, it seems you should let the user
run his MR job to generate HFiles however he sees fit, and then work on making
the {{completeBulkLoad}} portion able to transfer ownership.
It seems like a useful HDFS API to allow a {{chown}} call to work so long as
you have credentials for one user and a delegation token for another. We don't
have such a facility today, but I'm much more in support of adding that than
having hbase run user code under its own credentials.
> Secure Bulk Load
>
>
> Key: HBASE-5498
> URL: https://issues.apache.org/jira/browse/HBASE-5498
> Project: HBase
> Issue Type: Improvement
>Reporter: Francis Liu
>
> Design doc:
> https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load
> Short summary:
> Security as it stands does not cover the bulkLoadHFiles() feature. Users
> calling this method will bypass ACLs. Also loading is made more cumbersome in
> a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data
> from user's directory to the hbase directory, which would require certain
> write access privileges set.
> Our solution is to create a coprocessor which makes use of AuthManager to
> verify if a user has write access to the table. If so, launches a MR job as
> the hbase user to do the importing (ie rewrite from text to hfiles). One
> tricky part this job will have to do is impersonate the calling user when
> reading the input files. We can do this by expecting the user to pass an hdfs
> delegation token as part of the secureBulkLoad() coprocessor call and extend
> an inputformat to make use of that token. The output is written to a
> temporary directory accessible only by hbase and then bulkloadHFiles() is
> called.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4991) Provide capability to delete named region
[ https://issues.apache.org/jira/browse/HBASE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216138#comment-13216138 ] Todd Lipcon commented on HBASE-4991: Hey folks, sorry for being remiss in following this closely. Been a busy week! My thinking is that, while a FATE-like general framework would be nice, it's not a prerequisite for this feature. However, having this feature properly handle failure cases _is_ a prerequisite. I was thinking that, rather than handling it one-off for this feature, it might only be incrementally more work to do something like FATE and validate the new framework by developing this feature on top of it. It's a longer path to the end goal, but will result in something much more reusable for other similar features in the future (as well as some previous stuff being simplified). > Provide capability to delete named region > - > > Key: HBASE-4991 > URL: https://issues.apache.org/jira/browse/HBASE-4991 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Mubarak Seyed > Fix For: 0.94.0 > > Attachments: HBASE-4991.trunk.v1.patch, HBASE-4991.trunk.v2.patch > > > See discussion titled 'Able to control routing to Solr shards or not' on > lily-discuss > User may want to quickly dispose of out of date records by deleting specific > regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4991) Provide capability to delete named region
[ https://issues.apache.org/jira/browse/HBASE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216139#comment-13216139 ] Todd Lipcon commented on HBASE-4991: BTW, I'm also OK with the idea that the API be "deleteRange()" and the initial implementation be limited to only deleting an exact region, with the extension to automatically split and handle multiple regions becoming a followup. > Provide capability to delete named region > - > > Key: HBASE-4991 > URL: https://issues.apache.org/jira/browse/HBASE-4991 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Mubarak Seyed > Fix For: 0.94.0 > > Attachments: HBASE-4991.trunk.v1.patch, HBASE-4991.trunk.v2.patch > > > See discussion titled 'Able to control routing to Solr shards or not' on > lily-discuss > User may want to quickly dispose of out of date records by deleting specific > regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4365) Add a decent heuristic for region size
[ https://issues.apache.org/jira/browse/HBASE-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215088#comment-13215088 ] Todd Lipcon commented on HBASE-4365: Great results! Very cool. > Add a decent heuristic for region size > -- > > Key: HBASE-4365 > URL: https://issues.apache.org/jira/browse/HBASE-4365 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.1, 0.94.0 >Reporter: Todd Lipcon >Priority: Critical > Labels: usability > Attachments: 4365-v2.txt, 4365.txt > > > A few of us were brainstorming this morning about what the default region > size should be. There were a few general points made: > - in some ways it's better to be too-large than too-small, since you can > always split a table further, but you can't merge regions currently > - with HFile v2 and multithreaded compactions there are fewer reasons to > avoid very-large regions (10GB+) > - for small tables you may want a small region size just so you can > distribute load better across a cluster > - for big tables, multi-GB is probably best -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5347) GC free memory management in Level-1 Block Cache
[
https://issues.apache.org/jira/browse/HBASE-5347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214417#comment-13214417
]
Todd Lipcon commented on HBASE-5347:
Rather than introduce a global config, each filter could use an annotation or
marker (ie empty) interface, eg:
{code}
class MyFilter extends FilterBase implements RefCountAware
{code}
or
{code}
@RefCountAware
class MyFilter extends FilterBase
{code}
> GC free memory management in Level-1 Block Cache
>
>
> Key: HBASE-5347
> URL: https://issues.apache.org/jira/browse/HBASE-5347
> Project: HBase
> Issue Type: Improvement
>Reporter: Prakash Khemani
>Assignee: Prakash Khemani
> Attachments: D1635.5.patch
>
>
> On eviction of a block from the block-cache, instead of waiting for the
> garbage collecter to reuse its memory, reuse the block right away.
> This will require us to keep reference counts on the HFile blocks. Once we
> have the reference counts in place we can do our own simple
> blocks-out-of-slab allocation for the block-cache.
> This will help us with
> * reducing gc pressure, especially in the old generation
> * making it possible to have non-java-heap memory backing the HFile blocks
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3484) Replace memstore's ConcurrentSkipListMap with our own implementation
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214179#comment-13214179 ] Todd Lipcon commented on HBASE-3484: bq. How you think Todd? Because of the tiering cost more or is it something to do w/ mslab allocations? Extra container costs with the fact that we have extra CSLM objects for each row. I haven't measured but I bet there is some map-wide overhead that we're paying. There are some other things I noticed that could be improved, though. In particular, CSLM optimizes for Comparable keys, so if you specify a custom comparator, then it has to wrap every key you insert with a wrapper object. Specializing CSLM for our purposes would easily save 64 bytes per entry on this. Another thought I had was to do the following: - have the actual entries in rowMap be Object-typed, rather than CSLMs. - when the first insert happens, just insert the KeyValue itself (optimization for the case where each row has only one cell) - when more inserts happen, swap it out for a proper container type The proper container type's also interesting to consider here. We never have contention on update within a row, since the updates happen under a row lock, right? So, we can consider any map type that supports single-writer multiple-reader efficiently, which is a wider range of data structures than support multi-writer multi-reader. One possibility is snap trees or even copy-on-write sorted array lists. bq. What would you like to see test-wise proving this direction better than what we currently have? I could work up some tests? Would be great if we had a benchmark focused on memstore-only which allowed a mix of the following operations from different threads: - full scans - range scans - updates to existing rows which just touch 1 or a few columns - updates to existing rows which touch lots of columns - inserts of new rows (few or lots of columns) But it's a bit of work to do all that. So, a microbenchmark which just timed something like having 20 threads each do a bunch of inserts with multi-column rows would at least show whether there's promise here. > Replace memstore's ConcurrentSkipListMap with our own implementation > > > Key: HBASE-3484 > URL: https://issues.apache.org/jira/browse/HBASE-3484 > Project: HBase > Issue Type: Improvement > Components: performance >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Priority: Critical > Attachments: hierarchical-map.txt > > > By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements > to it for our use case in MemStore: > - add an iterator.replace() method which should allow us to do upsert much > more cheaply > - implement a Set directly without having to do Map to > save one reference per entry > It turns out CSLM is in public domain from its development as part of JSR > 166, so we should be OK with licenses. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4365) Add a decent heuristic for region size
[ https://issues.apache.org/jira/browse/HBASE-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214147#comment-13214147 ] Todd Lipcon commented on HBASE-4365: OK :) > Add a decent heuristic for region size > -- > > Key: HBASE-4365 > URL: https://issues.apache.org/jira/browse/HBASE-4365 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.1, 0.94.0 >Reporter: Todd Lipcon >Priority: Critical > Labels: usability > Attachments: 4365-v2.txt, 4365.txt > > > A few of us were brainstorming this morning about what the default region > size should be. There were a few general points made: > - in some ways it's better to be too-large than too-small, since you can > always split a table further, but you can't merge regions currently > - with HFile v2 and multithreaded compactions there are fewer reasons to > avoid very-large regions (10GB+) > - for small tables you may want a small region size just so you can > distribute load better across a cluster > - for big tables, multi-GB is probably best -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4365) Add a decent heuristic for region size
[ https://issues.apache.org/jira/browse/HBASE-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214129#comment-13214129 ] Todd Lipcon commented on HBASE-4365: Got any comparison numbers for total import time, for say 100G load? Would be good to know that the new heuristic is definitely advantageous. > Add a decent heuristic for region size > -- > > Key: HBASE-4365 > URL: https://issues.apache.org/jira/browse/HBASE-4365 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.1, 0.94.0 >Reporter: Todd Lipcon >Priority: Critical > Labels: usability > Attachments: 4365-v2.txt, 4365.txt > > > A few of us were brainstorming this morning about what the default region > size should be. There were a few general points made: > - in some ways it's better to be too-large than too-small, since you can > always split a table further, but you can't merge regions currently > - with HFile v2 and multithreaded compactions there are fewer reasons to > avoid very-large regions (10GB+) > - for small tables you may want a small region size just so you can > distribute load better across a cluster > - for big tables, multi-GB is probably best -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5456) Introduce PowerMock into our unit tests to reduce unnecessary method exposure
[ https://issues.apache.org/jira/browse/HBASE-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214029#comment-13214029 ] Todd Lipcon commented on HBASE-5456: I tend to agree with Mikhail. The presence of a protected method named "getRegionServerServicesForTests" or something lets me know, when I'm working on the code, that this method is used, and I can easily use eclipse to tell me which unit tests use it. PowerMock and other tools which use strings to refer to functions aren't going to play nice with that, so it's easy to be unaware of test dependencies. I think these tools are best used sparingly and only to mock out system dependencies (like "new FileInputStream()", "InetSocketAddress.getHostName()", or "System.currentTimeMillis()") > Introduce PowerMock into our unit tests to reduce unnecessary method exposure > - > > Key: HBASE-5456 > URL: https://issues.apache.org/jira/browse/HBASE-5456 > Project: HBase > Issue Type: Task >Reporter: Zhihong Yu > > We should introduce PowerMock into our unit tests so that we don't have to > expose methods intended to be used by unit tests. > Here was Benoit's reply to a user of asynchbase about testability: > OpenTSDB has unit tests that are mocking out HBaseClient just fine > [1]. You can mock out pretty much anything on the JVM: final, > private, JDK stuff, etc. All you need is the right tools. I've been > very happy with PowerMock. It supports Mockito and EasyMock. > I've never been keen on mutilating public interfaces for the sake of > testing. With tools like PowerMock, we can keep the public APIs tidy > while mocking and overriding anything, even in the most private guts > of the classes. > [1] > https://github.com/stumbleupon/opentsdb/blob/master/src/uid/TestUniqueId.java#L66 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4991) Provide capability to delete named region
[ https://issues.apache.org/jira/browse/HBASE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212228#comment-13212228 ] Todd Lipcon commented on HBASE-4991: I'm cool with the API now. I'm a bit nervous about the complexity of the implementation - the fact that this (and the online schema change) have to do things like disable splits and disable balancing is a little "smelly" to me. But, I don't necessarily have a better idea. In terms of testing before commit, I'd like to see a kind of stress test which can be run continuously that does something like the following in a loop: - create a table with 10 regions - insert 10 rows into each region - pick a random order to delete the regions - start some threads which perform full table scans to count rows. concurrently, delete each of the regions in the above random order - number of rows should drop 100->90->80, etc. - drop table If we can run this test continuously, and the test can still succeed even if we periodically kill the master or one of the RSes, it seems sufficient. > Provide capability to delete named region > - > > Key: HBASE-4991 > URL: https://issues.apache.org/jira/browse/HBASE-4991 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Mubarak Seyed > Fix For: 0.94.0 > > Attachments: HBASE-4991.trunk.v1.patch, HBASE-4991.trunk.v2.patch > > > See discussion titled 'Able to control routing to Solr shards or not' on > lily-discuss > User may want to quickly dispose of out of date records by deleting specific > regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5286) bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when presented with split packaged Hadoop 0.23 installation
[
https://issues.apache.org/jira/browse/HBASE-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212223#comment-13212223
]
Todd Lipcon commented on HBASE-5286:
What's the advantage of using the hadoop binary to execute HBase instead of
doing something like:
{code}
HADOOP_CLASSPATH=$(hadoop classpath)
{code}
to use its classpath construction logic? This should work in any reasonably
recent Hadoop version.
We'd probably want to add something to Hadoop like {{hadoop librarypath}} to
get the native libs as well, but this seems a little better than actually using
the whole hadoop wrapper script.
> bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when
> presented with split packaged Hadoop 0.23 installation
>
>
> Key: HBASE-5286
> URL: https://issues.apache.org/jira/browse/HBASE-5286
> Project: HBase
> Issue Type: Bug
> Components: scripts
>Affects Versions: 0.92.0
>Reporter: Roman Shaposhnik
>Assignee: Roman Shaposhnik
>
> Here's the bit from bin/hbase that might need TLC now that Hadoop can be
> spotted in the wild in split-package configuration:
> {noformat}
> #If avail, add Hadoop to the CLASSPATH and to the JAVA_LIBRARY_PATH
> if [ ! -z $HADOOP_HOME ]; then
> HADOOPCPPATH=""
> if [ -z $HADOOP_CONF_DIR ]; then
> HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" "${HADOOP_HOME}/conf")
> else
> HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" "${HADOOP_CONF_DIR}")
> fi
> if [ "`echo ${HADOOP_HOME}/hadoop-core*.jar`" !=
> "${HADOOP_HOME}/hadoop-core*.jar" ] ; then
> HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls
> ${HADOOP_HOME}/hadoop-core*.jar | head -1`)
> else
> HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls
> ${HADOOP_HOME}/hadoop-common*.jar | head -1`)
> HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls
> ${HADOOP_HOME}/hadoop-hdfs*.jar | head -1`)
> HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls
> ${HADOOP_HOME}/hadoop-mapred*.jar | head -1`)
> fi
> {noformat}
> There's a couple of issues with the above code:
>0. HADOOP_HOME is now deprecated in Hadoop 0.23
>1. the list of jar files added to the class-path should be revised
>2. we need to figure out a more robust way to get the jar files that are
> needed to the classpath (things like hadoop-mapred*.jar tend to match
> src/test jars as well)
> Better yet, it would be useful to look into whether we can transition HBase's
> bin/hbase onto using bin/hadoop as a launcher script instead of direct JAVA
> invocations (Pig, Hive, Sqoop and Mahout already do that)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5241) Deletes should not mask Puts that come after it.
[
https://issues.apache.org/jira/browse/HBASE-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212214#comment-13212214
]
Todd Lipcon commented on HBASE-5241:
{quote}
Now that we are flushing the memstoreTS to disk, along with the KVs, we should
be able
to differentiate whether or not the Put happened after the Delete and offer
better
delete semantics.
{quote}
What are the "better semantics" that we would offer? ie, if I do:
- put value "a" at ts=1
- delete at ts=3
- put value "b" at ts=2
and I do a read with "current time" semantics, do you expect to see "b" or
nothing? I'm not convinced that "b" is a "better semantic" here, except for the
point that it makes major compaction more transparent. The transparency of
compaction is sort of nice, but compaction is already not transparent because
of time travel reads (except for the "always keep versions" stuff that we did
recently)
> Deletes should not mask Puts that come after it.
>
>
> Key: HBASE-5241
> URL: https://issues.apache.org/jira/browse/HBASE-5241
> Project: HBase
> Issue Type: Improvement
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Attachments: HBASE-5241.D1731.1.patch, HBASE-5241.D1731.2.patch,
> HBASE-5241.D1731.3.patch
>
>
> Suppose that we have a delete row, and then followed by the put. The delete
> row
> can mask the put, unless there was a major compaction in between.
> Now that we are flushing the memstoreTS to disk, along with the KVs, we
> should be able
> to differentiate whether or not the Put happened after the Delete and offer
> better
> delete semantics.
> Couldn't find a pre-existing JIRA that already discusses this, so creating
> one.
> Seems related to https://issues.apache.org/jira/browse/HBASE-2406, but is not
> quite the same.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5347) GC free memory management in Level-1 Block Cache
[ https://issues.apache.org/jira/browse/HBASE-5347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210750#comment-13210750 ] Todd Lipcon commented on HBASE-5347: Hey folks. I haven't been through the patch yet, but just wanted to throw out one idea that I think can make reference-counted systems a little simpler: in Cocoa (the OSX development framework) there's a class called NSAutoreleasePool, an instance of which is carried around as part of the local thread context. You can then call "autorelease" on any object, which will not immediately decrement the ref count, but adds it to the pool. When you release the pool, all referenced objects are decremented at that point. This idea might make it easier to manage references. For example, when something is read by a scanner, it could be read with ref count incremented but put on the request's autorelease pool. Then, when any IPC handler thread is returned to the thread pool, the auto release pool could be decremented. This ensures that any stuff we reference is kept around for the whole request lifecycle but still automatically dereffed at the end. Do you think such a construct would be useful here? https://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/MemoryMgmt/Articles/mmAutoreleasePools.html has some more info. > GC free memory management in Level-1 Block Cache > > > Key: HBASE-5347 > URL: https://issues.apache.org/jira/browse/HBASE-5347 > Project: HBase > Issue Type: Improvement >Reporter: Prakash Khemani >Assignee: Prakash Khemani > Attachments: D1635.5.patch > > > On eviction of a block from the block-cache, instead of waiting for the > garbage collecter to reuse its memory, reuse the block right away. > This will require us to keep reference counts on the HFile blocks. Once we > have the reference counts in place we can do our own simple > blocks-out-of-slab allocation for the block-cache. > This will help us with > * reducing gc pressure, especially in the old generation > * making it possible to have non-java-heap memory backing the HFile blocks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4991) Provide capability to delete named region
[ https://issues.apache.org/jira/browse/HBASE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209931#comment-13209931 ] Todd Lipcon commented on HBASE-4991: (btw, I am OK with the API that takes a key range and requires that the key range map to a region initially). But the above comments about implementation stand - can this be done in such a way that it shares code with "merge"? i.e that it's a two step process: step 1 is to clear the memstore + delete hfiles in a region (so the region still exists and is empty), and step 2 being that we initiate a merge to kill the now-empty one? The first step would then be RS-local, which makes it simpler, and the second step would cover other use cases as well > Provide capability to delete named region > - > > Key: HBASE-4991 > URL: https://issues.apache.org/jira/browse/HBASE-4991 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Mubarak Seyed > Fix For: 0.94.0 > > Attachments: HBASE-4991.trunk.v1.patch, HBASE-4991.trunk.v2.patch > > > See discussion titled 'Able to control routing to Solr shards or not' on > lily-discuss > User may want to quickly dispose of out of date records by deleting specific > regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4991) Provide capability to delete named region
[
https://issues.apache.org/jira/browse/HBASE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209924#comment-13209924
]
Todd Lipcon commented on HBASE-4991:
bq. Hmm, maybe I am a bit slow: in both cases the data of the named region
wouldn't exist any more on the original region server.
But region servers are also not part of the *data model*.
There's the implementation (region servers, regions, etc) and the data model
(rows, columns, etc). {{move}} is an administrative/operational thing (like
turning on/off load balancer or gathering metrics). {{delete...} is a data
model thing (it affects what's stored). Having APIs that mix the two is messy.
> Provide capability to delete named region
> -
>
> Key: HBASE-4991
> URL: https://issues.apache.org/jira/browse/HBASE-4991
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Mubarak Seyed
> Fix For: 0.94.0
>
> Attachments: HBASE-4991.trunk.v1.patch, HBASE-4991.trunk.v2.patch
>
>
> See discussion titled 'Able to control routing to Solr shards or not' on
> lily-discuss
> User may want to quickly dispose of out of date records by deleting specific
> regions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4991) Provide capability to delete named region
[
https://issues.apache.org/jira/browse/HBASE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209911#comment-13209911
]
Todd Lipcon commented on HBASE-4991:
The difference is that {{move}} is an operational thing affecting load
balancing. The data in the database stays the same. {{deleteRegion}} leaks the
concept of regions into the data model itself, since it affects what's stored.
> Provide capability to delete named region
> -
>
> Key: HBASE-4991
> URL: https://issues.apache.org/jira/browse/HBASE-4991
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Mubarak Seyed
> Fix For: 0.94.0
>
> Attachments: HBASE-4991.trunk.v1.patch, HBASE-4991.trunk.v2.patch
>
>
> See discussion titled 'Able to control routing to Solr shards or not' on
> lily-discuss
> User may want to quickly dispose of out of date records by deleting specific
> regions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4403) Adopt interface stability/audience classifications from Hadoop
[ https://issues.apache.org/jira/browse/HBASE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209798#comment-13209798 ] Todd Lipcon commented on HBASE-4403: We should also get the javadoc doclet set up so that we don't generate javadoc for the non-public APIs. > Adopt interface stability/audience classifications from Hadoop > -- > > Key: HBASE-4403 > URL: https://issues.apache.org/jira/browse/HBASE-4403 > Project: HBase > Issue Type: Task >Affects Versions: 0.90.5, 0.92.0 >Reporter: Todd Lipcon >Assignee: Jimmy Xiang > Attachments: hbase-4403-interface.txt, > hbase-4403-nowhere-near-done.txt > > > As HBase gets more widely used, we need to be more explicit about which APIs > are stable and not expected to break between versions, which APIs are still > evolving, etc. We also have many public classes that are really internal to > the RS or Master and not meant to be used by users. Hadoop has adopted a > classification scheme for audience (public, private, or limited-private) as > well as stability (stable, evolving, unstable). I think we should copy these > annotations to HBase and start to classify our public classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4403) Adopt interface stability/audience classifications from Hadoop
[ https://issues.apache.org/jira/browse/HBASE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209740#comment-13209740 ] Todd Lipcon commented on HBASE-4403: Oh, I should add to the above that we're allowed to *add* APIs to an API even if it's marked Stable. We just can't break old APIs. > Adopt interface stability/audience classifications from Hadoop > -- > > Key: HBASE-4403 > URL: https://issues.apache.org/jira/browse/HBASE-4403 > Project: HBase > Issue Type: Task >Affects Versions: 0.90.5, 0.92.0 >Reporter: Todd Lipcon >Assignee: Jimmy Xiang > Attachments: hbase-4403-interface.txt, > hbase-4403-nowhere-near-done.txt > > > As HBase gets more widely used, we need to be more explicit about which APIs > are stable and not expected to break between versions, which APIs are still > evolving, etc. We also have many public classes that are really internal to > the RS or Master and not meant to be used by users. Hadoop has adopted a > classification scheme for audience (public, private, or limited-private) as > well as stability (stable, evolving, unstable). I think we should copy these > annotations to HBase and start to classify our public classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4991) Provide capability to delete named region
[
https://issues.apache.org/jira/browse/HBASE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209666#comment-13209666
]
Todd Lipcon commented on HBASE-4991:
Note also the comment in the Lily thread referenced:
{quote}
He he, cool. Unfortunately we will not be able to use a HBase feature where you
can quickly delete named regions
{quote}
Maybe some others can explain their use cases here?
> Provide capability to delete named region
> -
>
> Key: HBASE-4991
> URL: https://issues.apache.org/jira/browse/HBASE-4991
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Mubarak Seyed
> Fix For: 0.94.0
>
> Attachments: HBASE-4991.trunk.v1.patch, HBASE-4991.trunk.v2.patch
>
>
> See discussion titled 'Able to control routing to Solr shards or not' on
> lily-discuss
> User may want to quickly dispose of out of date records by deleting specific
> regions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4991) Provide capability to delete named region
[ https://issues.apache.org/jira/browse/HBASE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209661#comment-13209661 ] Todd Lipcon commented on HBASE-4991: Maybe I'm misunderstanding the use case, but I'm slightly skeptical that there isn't a simpler way of solving the same thing. It seems like this conflates two operations: 1) delete a bunch of data within a range, and 2) merge a region back with its neighbors. #2 is a more general "merge small regions" operation, which I think should be attacked orthogonally to the issue of #1 (bulk range delete). My other concern is that the API here exposes the concept of regions. A better API would be deleteRange(startKey, endKey). It might be implemented underneath by splitting the table such that startKey is the first key in some region, and endKey is the last key in another, and then deleting the underlying regions. See http://incubator.apache.org/accumulo/user_manual_1.4-incubating/Table_Configuration.html#Delete_Range for example > Provide capability to delete named region > - > > Key: HBASE-4991 > URL: https://issues.apache.org/jira/browse/HBASE-4991 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Mubarak Seyed > Fix For: 0.94.0 > > Attachments: HBASE-4991.trunk.v1.patch, HBASE-4991.trunk.v2.patch > > > See discussion titled 'Able to control routing to Solr shards or not' on > lily-discuss > User may want to quickly dispose of out of date records by deleting specific > regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5402) PerformanceEvaluation creates the wrong number of rows in randomWrite
[
https://issues.apache.org/jira/browse/HBASE-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209550#comment-13209550
]
Todd Lipcon commented on HBASE-5402:
Why don't we change it so that, instead of random, it just counts from 1 to 1M
but puts the bits through some kind of blender? Easiest is to reverse the bit
order, but could also do more creative swapping. Each mapper could also take
its ascribed range of keys, break it into 100 subranges, and randomize each of
the subranges.
> PerformanceEvaluation creates the wrong number of rows in randomWrite
> -
>
> Key: HBASE-5402
> URL: https://issues.apache.org/jira/browse/HBASE-5402
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Oliver Meyn
>
> The command line 'hbase org.apache.hadoop.hbase.PerformanceEvaluation
> randomWrite 10' should result in a table with 10 * (1024 * 1024) rows (so
> 10485760). Instead what happens is that the randomWrite job reports writing
> that many rows (exactly) but running rowcounter against the table reveals
> only e.g 6549899 rows. A second attempt to build the table produced slightly
> different results (e.g. 6627689). I see a similar discrepancy when using 50
> instead of 10 clients (~35% smaller than expected).
> Further experimentation reveals that the problem is key collision - by
> removing the % totalRows in getRandomRow I saw a reduction in collisions
> (table was ~8M rows instead of 6.6M). Replacing the random row key with
> UUIDs instead of Integers solved the problem and produced exactly 10485760
> rows. But that makes the key size 16 bytes instead of the current 10, so I'm
> not sure that's an acceptable solution.
> Here's the UUID code I used:
> public static byte[] format(final UUID uuid) {
> long msb = uuid.getMostSignificantBits();
> long lsb = uuid.getLeastSignificantBits();
> byte[] buffer = new byte[16];
> for (int i = 0; i < 8; i++) {
> buffer[i] = (byte) (msb >>> 8 * (7 - i));
> }
> for (int i = 8; i < 16; i++) {
> buffer[i] = (byte) (lsb >>> 8 * (7 - i));
> }
> return buffer;
> }
> which is invoked within getRandomRow with
> return format(UUID.randomUUID());
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209132#comment-13209132 ] Todd Lipcon commented on HBASE-5074: Hey Dhruba, I didn't look at the new rev yet, but does it also do checksums on the HFile header itself? ie the parts of the HFile that don't fall inside any block? If not, we should continue to use the checksummed FS when we open the hfile. -Todd On Wed, Feb 15, 2012 at 9:55 PM, dhruba (Dhruba Borthakur) < > support checksums in HBase block cache > -- > > Key: HBASE-5074 > URL: https://issues.apache.org/jira/browse/HBASE-5074 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: D1521.1.patch, D1521.1.patch, D1521.2.patch, > D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, > D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch > > > The current implementation of HDFS stores the data in one block file and the > metadata(checksum) in another block file. This means that every read into the > HBase block cache actually consumes two disk iops, one to the datafile and > one to the checksum file. This is a major problem for scaling HBase, because > HBase is usually bottlenecked on the number of random disk iops that the > storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5403) Checkpoint the compressed HLog
[ https://issues.apache.org/jira/browse/HBASE-5403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208843#comment-13208843 ] Todd Lipcon commented on HBASE-5403: Another option is to address this at the "read" side, since the failure recovery is probably a rare case. If the log reader sees an error, it can record the offset, start over, and "skip" records until it gets back to the point where it wants to start reading again. If we expect these failure scenarios to be unlikely, maybe this is better? > Checkpoint the compressed HLog > -- > > Key: HBASE-5403 > URL: https://issues.apache.org/jira/browse/HBASE-5403 > Project: HBase > Issue Type: Improvement >Reporter: Liyin Tang >Assignee: Liyin Tang > > Let's assume that HBase replication can be based on replaying the HLog in the > replica cluster. > The replica process could be crash during the replay. Obviously, the replica > process need a way to start from the lastest check point in the HLog, even > the HLog is compressed. > So the proposal is to write a series of checkpoints within the HLog. > Each each checkpoint will have a header with some special sequence of bytes. > And between each checkpoints, HLog should use new dictionaries to compress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5353) HA/Distributed HMaster via RegionServers
[ https://issues.apache.org/jira/browse/HBASE-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208006#comment-13208006 ] Todd Lipcon commented on HBASE-5353: bq. Because of this, I think a more important feature is to have the backup masters setup a web server with an HTTP redirect to the active master's UI. +1 > HA/Distributed HMaster via RegionServers > > > Key: HBASE-5353 > URL: https://issues.apache.org/jira/browse/HBASE-5353 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 0.94.0 >Reporter: Jesse Yates >Priority: Minor > > Currently, the HMaster node(s) must be considered a 'special' node (though > not a single point of failover), meaning that the node must be protected more > than the other cluster machines or at least specially monitored. Minimally, > we always need to ensure that the master is running, rather than letting the > system handle that internally. It should be possible to instead have the > HMaster be much more available, either in a distributed sense (meaning a bit > rewrite) or multiple, dynamically created instances combined with the hot > fail-over of masters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5325) Expose basic information about the master-status through jmx beans
[ https://issues.apache.org/jira/browse/HBASE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205251#comment-13205251 ] Todd Lipcon commented on HBASE-5325: No, metrics2 is not in CDH, since it's neither compatible with earlier versions of metrics1 nor is it compatible with the metrics2 in 0.23+. > Expose basic information about the master-status through jmx beans > --- > > Key: HBASE-5325 > URL: https://issues.apache.org/jira/browse/HBASE-5325 > Project: HBase > Issue Type: Improvement >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Fix For: 0.94.0 > > Attachments: HBASE-5325.1.patch, HBASE-5325.wip.patch > > > Similar to the Namenode and Jobtracker, it would be good if the hbase master > could expose some information through mbeans. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5376) Add more logging to triage HBASE-5312: Closed parent region present in Hlog.lastSeqWritten
[ https://issues.apache.org/jira/browse/HBASE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205250#comment-13205250 ] Todd Lipcon commented on HBASE-5376: I bet we could also write a stress test - sounds like some race where we don't block off updates at the right spot when closing a region. Maybe a test which continually opens/closes a region while making edits, and after every close, verify it's not in the map? > Add more logging to triage HBASE-5312: Closed parent region present in > Hlog.lastSeqWritten > -- > > Key: HBASE-5376 > URL: https://issues.apache.org/jira/browse/HBASE-5376 > Project: HBase > Issue Type: Sub-task >Reporter: Jimmy Xiang >Priority: Minor > Fix For: 0.90.7 > > > It is hard to find out what exactly caused HBASE-5312. Some logging will be > helpful to shine some lights. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5373) Table level lock to prevent the race of multiple table level operation
[ https://issues.apache.org/jira/browse/HBASE-5373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205108#comment-13205108 ] Todd Lipcon commented on HBASE-5373: Over in Accumulo land they have a neat framework called "FATE" (FAult Tolerant Execution or something?) Basically, they put up a znode in ZK for any master-coordinated action, and it acts as an "intent log" of the idempotent steps. If there's a failover, the new master will finish off any already-running FATE transaction. > Table level lock to prevent the race of multiple table level operation > -- > > Key: HBASE-5373 > URL: https://issues.apache.org/jira/browse/HBASE-5373 > Project: HBase > Issue Type: Improvement >Reporter: Liyin Tang >Assignee: Liyin Tang > > A table level lock can guarantee that only one table operation would happen > at one time for each table. The master should require and release these table > locks correctly during the failover time. One proposal is to keep track of > the lock and its corresponding operation in the zookeeper. If there is a > master failover, the secondary should have a way to check whether these > operations are succeeded nor not before releasing the lock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5363) Add rat check to run automatically on mvn build.
[ https://issues.apache.org/jira/browse/HBASE-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204695#comment-13204695 ] Todd Lipcon commented on HBASE-5363: Can we enable this only with a -Prelease build? Or part of the assembly phase? The package phase runs for every "mvn install", which is already super slow.. I think we could move this and the source jar generation to a different phase, or selectively enable with a -Prelease profile? > Add rat check to run automatically on mvn build. > > > Key: HBASE-5363 > URL: https://issues.apache.org/jira/browse/HBASE-5363 > Project: HBase > Issue Type: Improvement > Components: build >Affects Versions: 0.90.5, 0.92.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > > Some of the recent hbase release failed rat checks (mvn rat:check). We > should add checks likely in the mvn package phase so that this becomes a > non-issue in the future. > Here's an example from Whirr: > https://github.com/apache/whirr/blob/trunk/pom.xml line 388 for an example. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5312) Closed parent region present in Hlog.lastSeqWritten
[
https://issues.apache.org/jira/browse/HBASE-5312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204313#comment-13204313
]
Todd Lipcon commented on HBASE-5312:
This bug has been around a while, see:
https://issues.apache.org/jira/browse/HBASE-3481?focusedCommentId=12987965&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12987965
We also experienced it on a customer cluster this week.
Any leads on this, Ram?
> Closed parent region present in Hlog.lastSeqWritten
> ---
>
> Key: HBASE-5312
> URL: https://issues.apache.org/jira/browse/HBASE-5312
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.5
>Reporter: ramkrishna.s.vasudevan
> Fix For: 0.90.7
>
>
> This is in reference to the mail sent in the dev mailing list
> "Closed parent region present in Hlog.lastSeqWritten".
> The sceanrio described is
> We had a region that was split into two daughters. When the hlog roll tried
> to flush the region there was an entry in the HLog.lastSeqWritten that was
> not flushed or removed from the lastSeqWritten during the parent close.
> Because this flush was not happening subsequent flushes were getting blocked
> {code}
> 05:06:44,422 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many
> hlogs: logs=122, maxlogs=32; forcing flush of 1 regions(s):
> 2acaf8e3acfd2e8a5825a1f6f0aca4a8
> 05:06:44,422 WARN org.apache.hadoop.hbase.regionserver.LogRoller: Failed
> to schedule flush of 2acaf8e3acfd2e8a5825a1f6f0aca4a8r=null,
> requester=null
> 05:10:48,666 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many
> hlogs: logs=123, maxlogs=32; forcing flush of 1 regions(s):
> 2acaf8e3acfd2e8a5825a1f6f0aca4a8
> 05:10:48,666 WARN org.apache.hadoop.hbase.regionserver.LogRoller: Failed
> to schedule flush of 2acaf8e3acfd2e8a5825a1f6f0aca4a8r=null,
> requester=null
> 05:14:46,075 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many
> hlogs: logs=124, maxlogs=32; forcing flush of 1 regions(s):
> 2acaf8e3acfd2e8a5825a1f6f0aca4a8
> 05:14:46,075 WARN org.apache.hadoop.hbase.regionserver.LogRoller: Failed
> to schedule flush of 2acaf8e3acfd2e8a5825a1f6f0aca4a8r=null,
> requester=null
> 05:15:41,584 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many
> hlogs: logs=125, maxlogs=32; forcing flush of 1 regions(s):
> 2acaf8e3acfd2e8a5825a1f6f0aca4a8
> 05:15:41,584 WARN org.apache.hadoop.hbase.regionserver.LogRoller: Failed
> to schedule flush of 2acaf8e3acfd2e8a5825a1f6f0aca4a8r=null,
> {code}
> Lets see what happened for the region 2acaf8e3acfd2e8a5825a1f6f0aca4a8
> {code}
> 2012-01-06 00:30:55,214 INFO org.apache.hadoop.hbase.regionserver.Store:
> Renaming flushed file at
> hdfs://192.168.1.103:9000/hbase/Htable_UFDR_031/2acaf8e3acfd2e8a5825a1f6f0aca4a8/.tmp/1755862026714756815
> to
> hdfs://192.168.1.103:9000/hbase/Htable_UFDR_031/2acaf8e3acfd2e8a5825a1f6f0aca4a8/value/973789709483406123
> 2012-01-06 00:30:58,946 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> Instantiated
> Htable_UFDR_016,049790700093168-0456520,1325809837958.0ebe5bd7fcbc09ee074d5600b9d4e062.
> 2012-01-06 00:30:59,614 INFO org.apache.hadoop.hbase.regionserver.Store:
> Added
> hdfs://192.168.1.103:9000/hbase/Htable_UFDR_031/2acaf8e3acfd2e8a5825a1f6f0aca4a8/value/973789709483406123,
> entries=7537, sequenceid=20312223, memsize=4.2m, filesize=2.9m
> 2012-01-06 00:30:59,787 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> Finished snapshotting, commencing flushing stores
> 2012-01-06 00:30:59,787 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Finished memstore flush of ~133.5m for region
> Htable_UFDR_031,00332,1325808823997.2acaf8e3acfd2e8a5825a1f6f0aca4a8. in
> 21816ms, sequenceid=20312223, compaction requested=true
> 2012-01-06 00:30:59,787 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested
> for Htable_UFDR_031,00332,1325808823997.2acaf8e3acfd2e8a5825a1f6f0aca4a8.
> because regionserver20020.cacheFlusher; priority=0, compaction queue size=5840
> {code}
> A user triggered split has been issued to this region which can be seen in
> the above logs.
> The flushing of this region has resulted in a seq id 20312223.
> The region has been splitted and the parent region has been closed
> {code}
> 00:31:12,607 INFO org.apache.hadoop.hbase.regionserver.SplitTransaction:
> Starting split of region
> Htable_UFDR_031,00332,1325808823997.2acaf8e3acfd2e8a5825a1f6f0aca4a8.
> 00:31:13,694 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing
> Htable_UFDR_031,00332,1325808823997.2acaf8e3acfd2e8a5825a1f6f0aca4a8.:
> disabling compactions & flushes
> 00:31:13,694 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates
> disabled for region
> Htable_UFDR_031,00332,1325808823997.2acaf8e
[jira] [Commented] (HBASE-5355) Compressed RPC's for HBase
[ https://issues.apache.org/jira/browse/HBASE-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204296#comment-13204296 ] Todd Lipcon commented on HBASE-5355: bq. Todd in your proposal above we'd convert KVs to your new form before putting payload on wire and then undo it on other side? Yep, something like that. But hopefully it's just reference twiddling and not any actual data copying. bq. Long time back Ryan tried to a custom compression before putting stuff on the wire and then undoing it on other end hoping it would help some w/ latency but he found it upped latency I remember that too, but if I remember correctly it was going through the whole codec stream interface, which necessitates extra copies, etc. There's a lot of room for optimization there by adding new APIs to the hadoop codec stuff that operate on direct byte buffers in-place. And doing the "application level compression" as described above should be faster as well. > Compressed RPC's for HBase > -- > > Key: HBASE-5355 > URL: https://issues.apache.org/jira/browse/HBASE-5355 > Project: HBase > Issue Type: Improvement > Components: ipc >Affects Versions: 0.89.20100924 >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan > > Some application need ability to do large batched writes and reads from a > remote MR cluster. These eventually get bottlenecked on the network. These > results are also pretty compressible sometimes. > The aim here is to add the ability to do compressed calls to the server on > both the send and receive paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5353) HA/Distributed HMaster via RegionServers
[ https://issues.apache.org/jira/browse/HBASE-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204123#comment-13204123 ] Todd Lipcon commented on HBASE-5353: bq. We have the above (mostly unsolved) problems now if you run with more than one master. bq. They should be doing this now, if multiple masters? Sort of - except when you have two masters, you just set up nagios alerts and metrics to point to both, and you only need to look two places if you have an issue. If you have no idea where the master is, you have to hunt around the cluster to find it. bq. If the master function were lightweight enough, it'd be kinda sweet having one daemon type only I'd think Except we'd still have multiple daemon types, logically, it's just that they'd be collocated inside the same process, making logs harder to de-interleave, etc. Plus, if your RS are are all collocated with TTs and heavily loaded, then I wouldn't want to see the master running on one of them. I'd rather just tell ops "these nodes run the important master daemons, please monitor them and any high utilization is problematic". > HA/Distributed HMaster via RegionServers > > > Key: HBASE-5353 > URL: https://issues.apache.org/jira/browse/HBASE-5353 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 0.94.0 >Reporter: Jesse Yates >Priority: Minor > > Currently, the HMaster node must be considered a 'special' node (single point > of failure), meaning that the node must be protected more than the other > commodity machines. It should be possible to instead have the HMaster be much > more available, either in a distributed sense (meaning a bit rewrite) or with > multiple instances and automatic failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5357) Use builder pattern in StoreFile, HFile, and HColumnDescriptor instantiation
[
https://issues.apache.org/jira/browse/HBASE-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204108#comment-13204108
]
Todd Lipcon commented on HBASE-5357:
+1!
> Use builder pattern in StoreFile, HFile, and HColumnDescriptor instantiation
>
>
> Key: HBASE-5357
> URL: https://issues.apache.org/jira/browse/HBASE-5357
> Project: HBase
> Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>
> We have five ways to create an HFile writer, two ways to create a StoreFile
> writer, and the sets of parameters keep changing, creating a lot of
> confusion, especially when porting patches across branches. The same thing is
> happening to HColumnDescriptor. I think we should move to a "builder pattern"
> solution, e.g.
> {code:java}
> HFileWriter w = HFile.getWriterBuilder(conf, )
> .setParameter1(value1)
> .setParameter2(value2)
> ...
> .instantiate();
> {code}
> Each parameter setter being on the same line will make merges/cherry-pick
> work properly, we will not have to even mention default parameters again, and
> we can eliminate a dozen impossible-to-remember constructors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5353) HA/Distributed HMaster via RegionServers
[ https://issues.apache.org/jira/browse/HBASE-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204058#comment-13204058 ] Todd Lipcon commented on HBASE-5353: bq. The cluster knows about it, so you can have a link on the webui to the master or any of the region servers And each of the potential masters publishes metrics to ganglia, so if you want to find the master metrics, you have to hunt around in the ganglia graphs for which master was active at that time? And any cron jobs or nagios alerts you write need to first call some HBase utility to find the active master's IP via ZK in order to get to it? bq. True, but if those masters fail over, then your cluster management needs to be aware enough of that to provision more, on different servers If you have two masters on separate racks, and you have any reasonable monitoring, then your ops team will restart or provision a new one when they fail. I've never ever heard of this kind of scenario being a major cause of downtime. The whole thing seems like a bad idea to me. I won't -1 but consider me -0.5 > HA/Distributed HMaster via RegionServers > > > Key: HBASE-5353 > URL: https://issues.apache.org/jira/browse/HBASE-5353 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 0.94.0 >Reporter: Jesse Yates >Priority: Minor > > Currently, the HMaster node must be considered a 'special' node (single point > of failure), meaning that the node must be protected more than the other > commodity machines. It should be possible to instead have the HMaster be much > more available, either in a distributed sense (meaning a bit rewrite) or with > multiple instances and automatic failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5311) Allow inmemory Memstore compactions
[ https://issues.apache.org/jira/browse/HBASE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204055#comment-13204055 ] Todd Lipcon commented on HBASE-5311: bq. should 'state' be closed before being updated? I don't think it matters from a correctness standpoint. If we close the old state before updating the state variable, then all concurrent accessors beginning at this point will sit and spin while any previously running accessors finish their work. If we set the new state first, then any new accessors can immediately proceed so we don't cause any "hiccup" in access. The semantics of this whole data structure are a little subtle though so we'll have to be careful to expose only a minimal API and clearly document where we might have strange causality relations, etc. > Allow inmemory Memstore compactions > --- > > Key: HBASE-5311 > URL: https://issues.apache.org/jira/browse/HBASE-5311 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl > Attachments: InternallyLayeredMap.java > > > Just like we periodically compact the StoreFiles we should also periodically > compact the MemStore. > During these compactions we eliminate deleted cells, expired cells, cells to > removed because of version count, etc, before we even do a memstore flush. > Besides the optimization that we could get from this, it should also allow us > to remove the special handling of ICV, Increment, and Append (all of which > use upsert logic to avoid accumulating excessive cells in the Memstore). > Not targeting this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5353) HA/Distributed HMaster via RegionServers
[ https://issues.apache.org/jira/browse/HBASE-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204048#comment-13204048 ] Todd Lipcon commented on HBASE-5353: bq. But this means you don't need to worry about where you run your master Except it opens a new can of worms: where do you find the master UI? how do you monitor your master if it moves around? how do you easily find the master logs when it could be anywhere in the cluster? If your goal is to automatically pick a system to run a master on, you could have your cluster management software do that, but I only see additional complexity being introduced if you add this to HBase proper. > HA/Distributed HMaster via RegionServers > > > Key: HBASE-5353 > URL: https://issues.apache.org/jira/browse/HBASE-5353 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 0.94.0 >Reporter: Jesse Yates >Priority: Minor > > Currently, the HMaster node must be considered a 'special' node (single point > of failure), meaning that the node must be protected more than the other > commodity machines. It should be possible to instead have the HMaster be much > more available, either in a distributed sense (meaning a bit rewrite) or with > multiple instances and automatic failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5355) Compressed RPC's for HBase
[
https://issues.apache.org/jira/browse/HBASE-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204046#comment-13204046
]
Todd Lipcon commented on HBASE-5355:
I've been mulling this over in the back of my head recently with regards to the
work just getting started on adding extensible RPC. Here are a few thoughts:
A lot of our current lack of efficiency can be dealt with by simply avoiding
multiple copies of the same data. The most egregious example: when we serialize
columns, we serialize each KeyValue indepedendently, even though they all share
the same row key.
One potential solution I've been thinking about:
- introduce the concept of a "constant pool" which is associated with an RPC
request or response. This pool would be serialized on the wire before the
actual request/response and might be encoded like:
{code}
{code}
Then in the actual serialization of KeyValues, etc, we would not write out the
data, but rather indexes into the constant pool.
The advantages to this kind of technique would be:
- pushing all of the data near each other in the packet would make any
compression more beneficial (rather than interleaving compressible user data
with less compressible encoded information)
- allows multiple parts of a request/response to reference the same byte arrays
(eg multiple columns referring to the same row key)
- allows zero-copy implementations even if we use protobufs to encode the
actual call/response
This idea might be orthogonal to the compression discussed above, but may be a
cheaper (CPU-wise) way of getting a similar effect.
> Compressed RPC's for HBase
> --
>
> Key: HBASE-5355
> URL: https://issues.apache.org/jira/browse/HBASE-5355
> Project: HBase
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 0.89.20100924
>Reporter: Karthik Ranganathan
>Assignee: Karthik Ranganathan
>
> Some application need ability to do large batched writes and reads from a
> remote MR cluster. These eventually get bottlenecked on the network. These
> results are also pretty compressible sometimes.
> The aim here is to add the ability to do compressed calls to the server on
> both the send and receive paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5353) HA/Distributed HMaster via RegionServers
[ https://issues.apache.org/jira/browse/HBASE-5353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204036#comment-13204036 ] Todd Lipcon commented on HBASE-5353: Given that we already have failover support for the master, I'm skeptical that adding any complexity here is a good idea. If you want to colocate masters and RS, you can simply run a master process on a few of your RS nodes, and basically have the same behavior. What's the compelling use case? The master is _not_ a SPOF since we already have hot failover support. > HA/Distributed HMaster via RegionServers > > > Key: HBASE-5353 > URL: https://issues.apache.org/jira/browse/HBASE-5353 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 0.94.0 >Reporter: Jesse Yates >Priority: Minor > > Currently, the HMaster node must be considered a 'special' node (single point > of failure), meaning that the node must be protected more than the other > commodity machines. It should be possible to instead have the HMaster be much > more available, either in a distributed sense (meaning a bit rewrite) or with > multiple instances and automatic failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5313) Restructure hfiles layout for better compression
[ https://issues.apache.org/jira/browse/HBASE-5313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204031#comment-13204031 ] Todd Lipcon commented on HBASE-5313: I'm curious what the expected compression gain would be. Has anyone tried "rearranging" an example of a production hfile block and recompressing to see the difference? My thinking is that typical LZ-based compression (eg snappy) uses a hashtable for common substring identification which is up to 16K entries or so. So I don't know that it would do a particularly better job with the common keys if they were all grouped at the front of the block - so long as the keyval pairs are less than a few hundred bytes apart, it should still find them OK. Of course the other gains (storing large values compressed in RAM for example) seem good. > Restructure hfiles layout for better compression > > > Key: HBASE-5313 > URL: https://issues.apache.org/jira/browse/HBASE-5313 > Project: HBase > Issue Type: Improvement > Components: io >Reporter: dhruba borthakur >Assignee: dhruba borthakur > > A HFile block contain a stream of key-values. Can we can organize these kvs > on the disk in a better way so that we get much greater compression ratios? > One option (thanks Prakash) is to store all the keys in the beginning of the > block (let's call this the key-section) and then store all their > corresponding values towards the end of the block. This will allow us to > not-even decompress the values when we are scanning and skipping over rows in > the block. > Any other ideas? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5311) Allow inmemory Memstore compactions
[ https://issues.apache.org/jira/browse/HBASE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203364#comment-13203364 ] Todd Lipcon commented on HBASE-5311: The assumption with the code I was writing was a single "flusher" thread -- which could be enforced by making the "flush/compact" stuff synchronized on a separate lock. The idea is basically to allow the common case to run lockless and push all the effort to whoever wants to make updates to the data structure. > Allow inmemory Memstore compactions > --- > > Key: HBASE-5311 > URL: https://issues.apache.org/jira/browse/HBASE-5311 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl > Attachments: InternallyLayeredMap.java > > > Just like we periodically compact the StoreFiles we should also periodically > compact the MemStore. > During these compactions we eliminate deleted cells, expired cells, cells to > removed because of version count, etc, before we even do a memstore flush. > Besides the optimization that we could get from this, it should also allow us > to remove the special handling of ICV, Increment, and Append (all of which > use upsert logic to avoid accumulating excessive cells in the Memstore). > Not targeting this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5311) Allow inmemory Memstore compactions
[ https://issues.apache.org/jira/browse/HBASE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202818#comment-13202818 ] Todd Lipcon commented on HBASE-5311: yep, I clearly meant hash % counts.length :) Like I said, the code isn't tested at all, and is probably incorrect in more subtle ways as well! It's a long way from being useful, just wanted to post it in case Dhruba wanted to take a look. > Allow inmemory Memstore compactions > --- > > Key: HBASE-5311 > URL: https://issues.apache.org/jira/browse/HBASE-5311 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl > Attachments: InternallyLayeredMap.java > > > Just like we periodically compact the StoreFiles we should also periodically > compact the MemStore. > During these compactions we eliminate deleted cells, expired cells, cells to > removed because of version count, etc, before we even do a memstore flush. > Besides the optimization that we could get from this, it should also allow us > to remove the special handling of ICV, Increment, and Append (all of which > use upsert logic to avoid accumulating excessive cells in the Memstore). > Not targeting this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5229) Explore building blocks for "multi-row" local transactions.
[ https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202815#comment-13202815 ] Todd Lipcon commented on HBASE-5229: Gotcha - I hadn't looked at the most recent patch. Seems reasonable enough. > Explore building blocks for "multi-row" local transactions. > --- > > Key: HBASE-5229 > URL: https://issues.apache.org/jira/browse/HBASE-5229 > Project: HBase > Issue Type: New Feature > Components: client, regionserver >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.0 > > Attachments: 5229-endpoint.txt, 5229-multiRow-v2.txt, > 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt, 5229.txt > > > HBase should provide basic building blocks for multi-row local transactions. > Local means that we do this by co-locating the data. Global (cross region) > transactions are not discussed here. > After a bit of discussion two solutions have emerged: > 1. Keep the row-key for determining grouping and location and allow efficient > intra-row scanning. A client application would then model tables as > HBase-rows. > 2. Define a prefix-length in HTableDescriptor that defines a grouping of > rows. Regions will then never be split inside a grouping prefix. > #1 is true to the current storage paradigm of HBase. > #2 is true to the current client side API. > I will explore these two with sample patches here. > > Was: > As discussed (at length) on the dev mailing list with the HBASE-3584 and > HBASE-5203 committed, supporting atomic cross row transactions within a > region becomes simple. > I am aware of the hesitation about the usefulness of this feature, but we > have to start somewhere. > Let's use this jira for discussion, I'll attach a patch (with tests) > momentarily to make this concrete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5229) Explore building blocks for "multi-row" local transactions.
[ https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202670#comment-13202670 ] Todd Lipcon commented on HBASE-5229: Also, since this extends the semantics exposed by HBase we should probably point it out widely on the dev list to make sure everyone agrees with the direction. > Explore building blocks for "multi-row" local transactions. > --- > > Key: HBASE-5229 > URL: https://issues.apache.org/jira/browse/HBASE-5229 > Project: HBase > Issue Type: New Feature > Components: client, regionserver >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.0 > > Attachments: 5229-endpoint.txt, 5229-multiRow-v2.txt, > 5229-multiRow.txt, 5229-seekto-v2.txt, 5229-seekto.txt, 5229.txt > > > HBase should provide basic building blocks for multi-row local transactions. > Local means that we do this by co-locating the data. Global (cross region) > transactions are not discussed here. > After a bit of discussion two solutions have emerged: > 1. Keep the row-key for determining grouping and location and allow efficient > intra-row scanning. A client application would then model tables as > HBase-rows. > 2. Define a prefix-length in HTableDescriptor that defines a grouping of > rows. Regions will then never be split inside a grouping prefix. > #1 is true to the current storage paradigm of HBase. > #2 is true to the current client side API. > I will explore these two with sample patches here. > > Was: > As discussed (at length) on the dev mailing list with the HBASE-3584 and > HBASE-5203 committed, supporting atomic cross row transactions within a > region becomes simple. > I am aware of the hesitation about the usefulness of this feature, but we > have to start somewhere. > Let's use this jira for discussion, I'll attach a patch (with tests) > momentarily to make this concrete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5311) Allow inmemory Memstore compactions
[ https://issues.apache.org/jira/browse/HBASE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202629#comment-13202629 ] Todd Lipcon commented on HBASE-5311: The latter -- the whole memstore would be "layered", I think -- it's basically like we do with "snapshot" right now, except that there are multiple layers of "snapshot" and we have a trickier way of doing it without locks. > Allow inmemory Memstore compactions > --- > > Key: HBASE-5311 > URL: https://issues.apache.org/jira/browse/HBASE-5311 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl > Attachments: InternallyLayeredMap.java > > > Just like we periodically compact the StoreFiles we should also periodically > compact the MemStore. > During these compactions we eliminate deleted cells, expired cells, cells to > removed because of version count, etc, before we even do a memstore flush. > Besides the optimization that we could get from this, it should also allow us > to remove the special handling of ICV, Increment, and Append (all of which > use upsert logic to avoid accumulating excessive cells in the Memstore). > Not targeting this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5311) Allow inmemory Memstore compactions
[ https://issues.apache.org/jira/browse/HBASE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202193#comment-13202193 ] Todd Lipcon commented on HBASE-5311: Apparently this technique is called read-copy-update and the concurrency primitive is an RCU lock. There might be a more clever way of implementing it in Java but it's already probably over-optimized! > Allow inmemory Memstore compactions > --- > > Key: HBASE-5311 > URL: https://issues.apache.org/jira/browse/HBASE-5311 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl > Attachments: InternallyLayeredMap.java > > > Just like we periodically compact the StoreFiles we should also periodically > compact the MemStore. > During these compactions we eliminate deleted cells, expired cells, cells to > removed because of version count, etc, before we even do a memstore flush. > Besides the optimization that we could get from this, it should also allow us > to remove the special handling of ICV, Increment, and Append (all of which > use upsert logic to avoid accumulating excessive cells in the Memstore). > Not targeting this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5311) Allow inmemory Memstore compactions
[
https://issues.apache.org/jira/browse/HBASE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202178#comment-13202178
]
Todd Lipcon commented on HBASE-5311:
I don't think so... funnily enough I was just taking a break from my usually
scheduled work and working on this.
I don't have much so far, but my design uses some lock-free tricks to safely
swap around internal pieces of the data structure. Here's a sketch of what I
was hacking on:
I created a lock free data structure tentatively named "GatedSection" which is
sort of like a lock/critical section but not quite. It has the following API:
{code}
/**
* Return true if the thread should be allowed to enter the critical section,
* false if any other thread has already closed the gate
*/
boolean enterSection();
/**
* Must be called by any thread that has successfully entered above
*/
void exitSection();
/**
* Disallows any new threads from entering the section (future calls to
* enterSection() will return false). Blocks until all other threads which
* are already in the section have called exitSection().
* Postcondition: no threads are in the section or will be able to enter it
* after this method returns.
*/
void closeGateAndFlushThreads();
{code}
If you're familiar with JVM internals, this is a little bit like how
'checkpoints' work.
I then created another data structure:
{code}
class State {
GatedSection gate;
List> mapLayers; // writes go to idx 0, reads merge all
}
{code}
and the overall datastructure has a {{volatile State state;}} member.
Operations which mutate the current state (eg put) then do something like:
- in a loop, try to grab a local copy of the State variable, and then enter its
gate, until successful.
- modify its top map layer
- exit the gate
A "flush" works as follows:
- step 1: create a new top layer for writes
-- create a new State object which is identical except a new concurrent map has
been inserted at the "top" of mapLayers
-- swap the volatile "state" pointer to the new state. At this point, writers
will be writing into _either_ of the top two layers
-- close the "gate". After the gate is closed, writers will only be writing to
the _top_ layer
- step 2: freeze the old top layer
-- create a new efficient immutable datastructure for mapLayers.get(1) (no
rush to complete this phase)
-- swap it into place by swapping out the volatile state object
A compaction works similarly to "step 2" above.
Might be worth using a technique like fractional cascading on the "mapLayers"
structure as well to avoid O(k log n) lookups with k layers.
> Allow inmemory Memstore compactions
> ---
>
> Key: HBASE-5311
> URL: https://issues.apache.org/jira/browse/HBASE-5311
> Project: HBase
> Issue Type: Improvement
>Reporter: Lars Hofhansl
>
> Just like we periodically compact the StoreFiles we should also periodically
> compact the MemStore.
> During these compactions we eliminate deleted cells, expired cells, cells to
> removed because of version count, etc, before we even do a memstore flush.
> Besides the optimization that we could get from this, it should also allow us
> to remove the special handling of ICV, Increment, and Append (all of which
> use upsert logic to avoid accumulating excessive cells in the Memstore).
> Not targeting this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5323) Need to handle assertion error while splitting log through ServerShutDownHandler by shutting down the master
[
https://issues.apache.org/jira/browse/HBASE-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199917#comment-13199917
]
Todd Lipcon commented on HBASE-5323:
If this is an actual error condition that can be seen in practice (rather than
indication of a failed programmer assumption) it should be changed to an
if/throw, not an assert. Most users don't have asserts enabled in production.
> Need to handle assertion error while splitting log through
> ServerShutDownHandler by shutting down the master
>
>
> Key: HBASE-5323
> URL: https://issues.apache.org/jira/browse/HBASE-5323
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.5
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.94.0, 0.90.7
>
> Attachments: HBASE-5323.patch
>
>
> We know that while parsing the HLog we expect the proper length from HDFS.
> In WALReaderFSDataInputStream
> {code}
> assert(realLength >= this.length);
> {code}
> We are trying to come out if the above condition is not satisfied. But if
> SSH.splitLog() gets this problem then it lands in the run method of
> EventHandler. This kills the SSH thread and so further assignment does not
> happen. If ROOT and META are to be assigned they cannot be.
> I think in this condition we abort the master by catching such exceptions.
> Please do suggest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5304) Pluggable split key policy
[ https://issues.apache.org/jira/browse/HBASE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198409#comment-13198409 ] Todd Lipcon commented on HBASE-5304: Oh, I see... indeed you're right. Nope, no objection (though I did not review the code, only the discussion here) > Pluggable split key policy > -- > > Key: HBASE-5304 > URL: https://issues.apache.org/jira/browse/HBASE-5304 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.0 > > Attachments: 5304-v4.txt > > > We need a way to specify custom policies to determine split keys. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
