Re: Better Javadoc HTableDescriptor

2011-10-01 Thread Akash Ashok
While I was doing this I found out that there is a method in HTableDescriptor called isMetaTable() which basically checks if the table is a catalog table and the name kind of misleading. I am planning to rename this method to isCatalogTable(). 1. Is it ok to do this renaming considering that this

Multiple WALs

2011-10-01 Thread Akash Ashok
Hey, I've see that setting writeToWAL(false) boosts up the writes like crazy. I was just thinking having MuiltipleWAL on HBase. I understand that this is a consideration in BigTable paper that a WAL per region is not used because it might result in a lot of disk seeks when there are large number

Build failed in Jenkins: HBase-TRUNK #2278

2011-10-01 Thread Apache Jenkins Server
See https://builds.apache.org/job/HBase-TRUNK/2278/ -- [...truncated 1653 lines...] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.94 sec Running org.apache.hadoop.hbase.TestGlobalMemStoreSize Tests run: 1, Failures: 0, Errors: 0,

Build failed in Jenkins: HBase-TRUNK #2277

2011-10-01 Thread Apache Jenkins Server
See https://builds.apache.org/job/HBase-TRUNK/2277/ -- [...truncated 1653 lines...] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.134 sec Running org.apache.hadoop.hbase.TestGlobalMemStoreSize Tests run: 1, Failures: 0, Errors: 0,

Re: Better Javadoc HTableDescriptor

2011-10-01 Thread Doug Meil
Be very careful about code-changes that causes compilation failures upon upgrade. Once in a while that may happen on a major upgrade, but it had better be for a very good, explainable, reason that the community is behind. A fourth option is to add Javadoc to the explaining what this method

Re: Multiple WALs

2011-10-01 Thread Dhruba Borthakur
I have been experimenting with the WAL settings too. It is obvious that turning off the wal makes ur transactions go faster, HDFS write/sync are not yet very optimized for high throughput small writes. However, irrespective of whether I have one wal or two, I have seeing the same throughput. I

Re: Better Javadoc HTableDescriptor

2011-10-01 Thread Akash Ashok
I totally agree that its not at all accepted if it causes compilation failures on upgrades. But the very name of the function is very misleading considering the fact that there is a .META. region and its very hard to distinguish between isMetaRegion() and isMetaTable(). We could definitely add

Re: Better Javadoc HTableDescriptor

2011-10-01 Thread Ted Yu
I prefer the fourth option. Only truly broken APIs should be removed. Cheers On Oct 1, 2011, at 10:02 AM, Doug Meil doug.m...@explorysmedical.com wrote: Be very careful about code-changes that causes compilation failures upon upgrade. Once in a while that may happen on a major upgrade,

Re: Better Javadoc HTableDescriptor

2011-10-01 Thread Dhruba Borthakur
+1 for Option 4. -dhruba On Sat, Oct 1, 2011 at 11:26 AM, Ted Yu yuzhih...@gmail.com wrote: I prefer the fourth option. Only truly broken APIs should be removed. Cheers On Oct 1, 2011, at 10:02 AM, Doug Meil doug.m...@explorysmedical.com wrote: Be very careful about code-changes

Re: Multiple WALs

2011-10-01 Thread Stack
+1 on making WAL pluggable so we can experiment. Being able to write multiple WALs at once should be easy enough to do (the WAL split code should be able to handle it). Also a suggestion made a while back was making it so hbase could be configured to write two filesystems -- there'd be

Re: Better Javadoc HTableDescriptor

2011-10-01 Thread Akash Ashok
Ok Thanks. I shall just add the javadoc. I've already opened a JIRA for improving the javadoc https://issues.apache.org/jira/browse/HBASE-4486 Cheers, Akash A On Sun, Oct 2, 2011 at 12:21 AM, Dhruba Borthakur dhr...@gmail.com wrote: +1 for Option 4. -dhruba On Sat, Oct 1, 2011 at 11:26

Re: Multiple WALs

2011-10-01 Thread karthik tunga
Hey, Doesn't multiple WALs need some kind of merging when recovering from a crash ? Cheers, Karthik On 1 October 2011 15:17, Stack st...@duboce.net wrote: +1 on making WAL pluggable so we can experiment. Being able to write multiple WALs at once should be easy enough to do (the WAL split

Re: Better Javadoc HTableDescriptor

2011-10-01 Thread Shumin Wu
Hello All, I am new here. So very nice to meet you guys. Here are my thoughts on this issue. I agree with Akash that the name of this method would confuse the user as there is a .META. notion that means something else. So why don't we create a function alias for isMetaTable() as a temporary

Re: Multiple WALs

2011-10-01 Thread Stack
Yes. For sure. Would need to check that the split can deal w/ multiple logs written by the one server concurrently (sort by sequence edit id after sorting on all the rest that makes up a wal log key). St.Ack On Sat, Oct 1, 2011 at 1:36 PM, karthik tunga karthik.tu...@gmail.com wrote: Hey,

Re: Multiple WALs

2011-10-01 Thread Jesse Yates
I think adding the abstraction layer and making it not only pluggable, but configurable would be great. It would be nice to be able to tie into a service that logs directly to disk, rather than go through HDFS giving some potentially awesome speedup at the cost of having to write a logging

Bay Area HBase User Group organizer change (?)

2011-10-01 Thread Andrew Purtell
I went to RSVP for the upcoming HUG in NYC after Hadoop World. Meetup complained the Bay Area HBase User Group was missing an organizer (who pays dues), and would be deleted after 14 days. I've paid up for us for the next 6 months, and am now the organizer. I'll figure out what is required of

Re: Bay Area HBase User Group organizer change (?)

2011-10-01 Thread Todd Lipcon
Thanks, Andrew! Let us know if we can chip in for the dues. -Todd On Sat, Oct 1, 2011 at 4:38 PM, Andrew Purtell apurt...@apache.org wrote: I went to RSVP for the upcoming HUG in NYC after Hadoop World. Meetup complained the Bay Area HBase User Group was missing an organizer (who pays

Re: Bay Area HBase User Group organizer change (?)

2011-10-01 Thread Ted Yu
I agree. We should share the payment. On Sat, Oct 1, 2011 at 5:05 PM, Todd Lipcon t...@cloudera.com wrote: Thanks, Andrew! Let us know if we can chip in for the dues. -Todd On Sat, Oct 1, 2011 at 4:38 PM, Andrew Purtell apurt...@apache.org wrote: I went to RSVP for the upcoming HUG in

Re: Multiple WALs

2011-10-01 Thread karthik tunga
Hey Stack, Along with the log replaying part, logic is also needed for log roll over. This, I think, easier compared to the merging of the logs. Any edits less than the last sequence number on the file system can be removed from all the WALs. Cheers, Karthik On 1 October 2011 18:05, Jesse

Re: Bay Area HBase User Group organizer change (?)

2011-10-01 Thread Ted Dunning
I can get some sponsorship going on my end as well. On Sun, Oct 2, 2011 at 12:09 AM, Ted Yu yuzhih...@gmail.com wrote: I agree. We should share the payment. On Sat, Oct 1, 2011 at 5:05 PM, Todd Lipcon t...@cloudera.com wrote: Thanks, Andrew! Let us know if we can chip in for the dues.

Re: Bay Area HBase User Group organizer change (?)

2011-10-01 Thread Ryan Rawson
Good job Andrew. Don't forget to expense it - problem solved! -ryan On Sat, Oct 1, 2011 at 6:26 PM, Ted Dunning tdunn...@maprtech.com wrote: I can get some sponsorship going on my end as well. On Sun, Oct 2, 2011 at 12:09 AM, Ted Yu yuzhih...@gmail.com wrote: I agree. We should share the

Re: Multiple WALs

2011-10-01 Thread Akash Ashok
I've opened up a JIRA for this https://issues.apache.org/jira/browse/HBASE-4529 Cheers, Akash A On Sun, Oct 2, 2011 at 6:04 AM, karthik tunga karthik.tu...@gmail.comwrote: Hey Stack, Along with the log replaying part, logic is also needed for log roll over. This, I think, easier compared to

Re: Multiple WALs

2011-10-01 Thread Akash Ashok
Uh Oh! Its showing the reporter of the problem as me but it wasn't me in reality :) I am not able to modify it. Please feel free to change it :) Cheers, Akash A On Sun, Oct 2, 2011 at 10:38 AM, Akash Ashok thehellma...@gmail.com wrote: I've opened up a JIRA for this