[hbase-5487] Master design discussion.

2013-10-19 Thread Jonathan Hsieh
sitions correspond to? Short doc: nit: Design Constraints, code should: Have AM logic isolated from the persistent storage of state. // I think this should be "abstracted" so we can plug in different implementations of persistent storage of state. -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: [hbase-5487] Master design discussion.

2013-10-19 Thread Jonathan Hsieh
we ensure that RS is dead we don't reassign. My document implies (but doesn't elaborate, I'll fix that) that master does direct Opened->Closed direct transition only when that is true. A state called "MaybeOpened" could be added. Let me add it... -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: [hbase-5487] Master design discussion.

2013-10-19 Thread Jonathan Hsieh
Responses inline. I'm going to wait for next doc before I ask more questions about the master5 specifics. :) I suggest we start a thread about the hang/double assign/fencing scenario. Jon. On Sat, Oct 19, 2013 at 7:17 AM, Jonathan Hsieh wrote: > Here's sergey's replies

Re: DISCUSSION: Move to GIT?

2013-10-25 Thread Jonathan Hsieh
ry and could still use features like > > >> git-bisect. > > >> > The result of that is at [2]. > > >> > > > >> > Joe Stein also sent us Kafka's use of Git that was helpful when > > writing > > >> > down how users should contribute patches. [3] > > >> > > > >> > I hope all this helps! Despite the pain we went through in getting > > this > > >> all > > >> > set up (dealing with coordination of INFRA to lock the SVN repos, > > >> transfer > > >> > to Git, verify accuracy, and people who then didn't know how to use > > >> Git), I > > >> > think everyone is happy we did it. > > >> > > > >> > - Josh > > >> > > > >> > [1] https://issues.apache.org/**jira/browse/INFRA-6392< > > >> > https://issues.apache.org/jira/browse/INFRA-6392> > > >> > [2] http://accumulo.apache.org/**git.html< > > >> > http://accumulo.apache.org/git.html> > > >> > [3] > > https://cwiki.apache.org/**confluence/display/KAFKA/Git+**Workflow< > > >> > https://cwiki.apache.org/confluence/display/KAFKA/Git+Workflow> > > >> > [4] https://git-wip-us.apache.org/ > > >> > > > >> > > >> > > >> -- > > >> --- > > >> Jesse Yates > > >> @jesse_yates > > >> jyates.github.com > > >> > > > > > > > > -- > > Best Regards, > > Haosdent Huang > > > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: jars in HBase 0.96

2013-11-13 Thread Jonathan Hsieh
l me what these jars are used for in HBase 0.96? > > If I remove them from lib dir, will it cause problems? > > > > Thanks, > > > > Jerry > > > > gmbal-api-only-3.0.0-b023.jar > > grizzly-framework-2.1.1.jar > > grizzly-framework-2.1.1-tests.jar >

Re: Calling o/s.flush() in HLog.sync()?

2013-11-14 Thread Jonathan Hsieh
ream > > (using Writables#write), and invoke sync/hflush. > > > > > https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java#L1314 > > > > Is there a reason we have this call here? Please let me know if I miss > any > > context. > > > > Thanks, > > Himanshu > > > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Adding frameworks and coding for testabilitywas [Re: Testing the fix for race condition between Compaction and StoreScanner.init]

2013-11-20 Thread Jonathan Hsieh
R_SEEK hook can be activated at the end of ctor of StoreScanner > wrapper created in #2 > The custom ChangedReadersObserver would activate the COMPACT_COMPLETE hook. > > Comments are welcome. > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: Adding frameworks and coding for testabilitywas [Re: Testing the fix for race condition between Compaction and StoreScanner.init]

2013-11-26 Thread Jonathan Hsieh
won't > need fault injection framework for commit tests.* > > +1. bolding mine. -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: Testing the fix for race condition between Compaction and StoreScanner.init

2013-12-02 Thread Jonathan Hsieh
> > wrapper created in #2 > > The custom ChangedReadersObserver would activate the COMPACT_COMPLETE > hook. > > > > Comments are welcome. > > > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: Testing the fix for race condition between Compaction and StoreScanner.init

2013-12-02 Thread Jonathan Hsieh
toBytes("cf"); + bq. Can you add enough info so that we don't need to consult the jira to > figure out what the unit test is testing? > > Sure. I will add comments to the new test. > Looking forwards to it. > Cheers > > > On Mon, Dec 2, 2013 at 11:06 AM,

Re: Testing the fix for race condition between Compaction and StoreScanner.init

2013-12-02 Thread Jonathan Hsieh
On Mon, Dec 2, 2013 at 7:34 PM, Ted Yu wrote: > bq. The lastest submitted trunk patch > > The proposal here wouldn't use minicluster. > > Great! > Cheers > > > On Mon, Dec 2, 2013 at 7:31 PM, Jonathan Hsieh wrote: > > > On Mon, Dec 2, 2013 at 1:13 PM,

[Shadow Regions / Read Replicas ]

2013-12-02 Thread Jonathan Hsieh
10070 -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: [Shadow Regions / Read Replicas ] Block Affinity

2013-12-02 Thread Jonathan Hsieh
hfds affinity group/favored nodes. So the shadow memstore for region X would flush the hfile to A,B,C and region Y to A,D,E. Then the read replicas would be promoted (close secondary, open as primary) based on where the regions/hfiles's affinity group. This feels likes an optimization done one the 2rd or 3rd rev. Jon -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: [Shadow Regions / Read Replicas ]

2013-12-02 Thread Jonathan Hsieh
ng sure the proposed design have a path for optimal fast-consistent read-recovery. Jon. On Mon, Dec 2, 2013 at 9:54 PM, Jonathan Hsieh wrote: > HBASE-10070 [1] looks to be heading into a discussion more apt for the > mailing list than in the jira. Moving this to the dev list for threaded

Re: [Shadow Regions / Read Replicas ] Wal per region?

2013-12-02 Thread Jonathan Hsieh
> Deveraj: > Jonathan Hsieh, WAL per region (WALpr) would give you the locality (and hence HDFS short > circuit) of reads if you were to couple it with the favored nodes. The cost is of course more WAL > files... In the current situation (no WALpr) it would create quite some traffic c

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Jonathan Hsieh
*. Can we agree on nomenclature? On Tue, Dec 3, 2013 at 11:07 AM, Enis Söztutar wrote: > Thanks Jon for bringing this to dev@. > > > On Mon, Dec 2, 2013 at 10:01 PM, Jonathan Hsieh wrote: > > > Fundamentally, I'd prefer focusing on making HBase "HBasier"

Re: [Shadow Regions / Read Replicas ] Wal per region?

2013-12-03 Thread Jonathan Hsieh
On Tue, Dec 3, 2013 at 11:21 AM, Devaraj Das wrote: > On Mon, Dec 2, 2013 at 10:20 PM, Jonathan Hsieh wrote: > > > > With this in mind, I actually I making the case that we would group the > all > > the regions from RS-A onto the same set of preferred regions servers.

Re: [Shadow Regions / Read Replicas ] Wal per region?

2013-12-03 Thread Jonathan Hsieh
On Tue, Dec 3, 2013 at 11:42 AM, Enis Söztutar wrote: > On Mon, Dec 2, 2013 at 10:20 PM, Jonathan Hsieh wrote: > > > > Deveraj: > > > Jonathan Hsieh, WAL per region (WALpr) would give you the locality (and > > hence HDFS short > > > circuit) of reads if

Re: [Shadow Regions / Read Replicas ] Block Affinity

2013-12-03 Thread Jonathan Hsieh
On Tue, Dec 3, 2013 at 11:37 AM, Enis Söztutar wrote: > Responses inlined. > > On Mon, Dec 2, 2013 at 10:00 PM, Jonathan Hsieh wrote: > > > For the most efficient consistent read-recovery (shadow > regions/memstores), > > it would make sense to have them assigned to t

Re: [Shadow Regions / Read Replicas ] Wal per region?

2013-12-03 Thread Jonathan Hsieh
On Tue, Dec 3, 2013 at 3:07 PM, Enis Söztutar wrote: > On Tue, Dec 3, 2013 at 2:03 PM, Jonathan Hsieh wrote: > > > On Tue, Dec 3, 2013 at 11:42 AM, Enis Söztutar > wrote: > > > > > On Mon, Dec 2, 2013 at 10:20 PM, Jonathan Hsieh > > wrote: > > >

Re: [Shadow Regions / Read Replicas ] Block Affinity

2013-12-03 Thread Jonathan Hsieh
be different potential sweet spots if read-replica regions and shadow memstore regions were co-located at region on recover time with trade offs for fast consistent recovery, ability to have more recent values, locality optimizations and load balancing optimizations. Jon. -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Jonathan Hsieh
x27;s master is based of the hbase 0.20/0.89 master which is significantly different than the hbase master from in 0.94/0.96/trunk today. Jon. -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: Extensions for patches accepted by QA bot

2013-12-03 Thread Jonathan Hsieh
; > > > > > > > > > > > > > > > > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: [Shadow Regions / Read Replicas ]

2013-12-03 Thread Jonathan Hsieh
On Tue, Dec 3, 2013 at 2:04 PM, Enis Söztutar wrote: > On Tue, Dec 3, 2013 at 11:51 AM, Jonathan Hsieh wrote:> > > > > On Tue, Dec 3, 2013 at 11:07 AM, Enis Söztutar wrote: > > > > > Thanks Jon for bringing this to dev@. > > > > > > &g

Re: [Shadow Regions / Read Replicas ] External replication disqualified?

2013-12-03 Thread Jonathan Hsieh
on against two clusters in the same HDFS / datacenter and add a "bulk load replication" feature? We'd get latency in the seconds (closer to my expected definition of eventual consistency) Jon On Tue, Dec 3, 2013 at 6:47 PM, Jonathan Hsieh wrote: > > > On Tue, Dec 3, 2013 at

Re: ANN: The second hbase-0.96-1 release candidate is available for download

2013-12-04 Thread Jonathan Hsieh
HBASE-9827 Intermittent TestLogRollingNoCluster#testContendedLogRolling > > failure > > HBASE-9785 Fix heap size reporting in HRegion > > HBASE-9821 Scanner id could collide > > HBASE-9694 HBaseTestingUtility.createTable infinite loop > > HBASE-9772 Normalize new client default values > > HBASE-9768 Two issues in AsyncProcess > > HBASE-9784 Switch to Hadoop-2.2 > > HBASE-9793 Offline a region before it's closed could cause double > > assignment > > HBASE-9798 Include dependency hamcrest-core > > HBASE-9796 npe in RegionServerCallable > > HBASE-9776 Test Load And Verify Fails with TableNotEnabledException > > HBASE-9788 [WINDOWS] Update rest server class name > > HBASE-9755 Cannot run classes in hbase-server tests jar from command line > > HBASE-9791 MR initializes scanner twice > > HBASE-9773 Master aborted when hbck asked the master to assign a region > > that was already online > > HBASE-9720 > > TestSplitTransactionOnCluster#testShutdownFixupWhenDaughterHasSplit > > occasionally times out > > HBASE-9771 [WebUI] Humanize store and blockcache statistics on RS > > HBASE-9570 With AccessDeniedException, HBase shell would be better to > just > > display the error message to be user friendly > > HBASE-9758 Log missing rows and their regions if ITBLL fails > > HBASE-9692 TestOrderedBytes doesn't display the arrays correctly > > > > > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: ANN: The second hbase-0.96-1 release candidate is available for download

2013-12-04 Thread Jonathan Hsieh
I don't know -- I'll test this against 0.96.0 and report. Jon. On Wed, Dec 4, 2013 at 8:13 AM, Jean-Marc Spaggiari wrote: > Hi Jon, > > Is your test running well on 0.96.0? > > JM > > > 2013/12/4 Jonathan Hsieh > > > -1 > > > > I ha

Re: ANN: The second hbase-0.96-1 release candidate is available for download

2013-12-04 Thread Jonathan Hsieh
I've filed an issue for this and will post more results there. (against 0.96.0 etc). https://issues.apache.org/jira/browse/HBASE-10079 On Wed, Dec 4, 2013 at 8:10 AM, Jonathan Hsieh wrote: > -1 > > I have a new test I tried on 0.96.1rc1. (polishing to share). Run #2 with

Re: ANN: The second hbase-0.96-1 release candidate is available for download

2013-12-04 Thread Jonathan Hsieh
peration specifically). > > > > ____ > From: Jonathan Hsieh > To: "dev@hbase.apache.org" > Sent: Wednesday, December 4, 2013 8:16 AM > Subject: Re: ANN: The second hbase-0.96-1 release candidate is available > for download > > > I've filed an

We haven't had a clean hadoopqa build since 10/2/13.

2013-12-11 Thread Jonathan Hsieh
) or even better just fix the problem. Thanks, Jon. -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: We haven't had a clean hadoopqa build since 10/2/13.

2013-12-11 Thread Jonathan Hsieh
> I do agree with Jon that no new warnings should be introduced. > > Currently 3 new Findbugs appear in QA reports. > > Cheers > > > On Wed, Dec 11, 2013 at 7:13 AM, Jonathan Hsieh > > > wrote: > > > Just a reminder -- we do not want to introduce new jav

Re: Hadoop version trunk dependency?

2013-12-11 Thread Jonathan Hsieh
> > > On Mon, Nov 11, 2013 at 12:31 PM, Stack wrote: > > > > > > > > > Should we at least deprecate it in 0.98 or maybe > > > > > even just drop it in 0.98? > > > > > > > > > > > > > +1 deprecate > > > > +0 drop > > > > > > > > -- > > > > Best regards, > > > > > > > >- Andy > > > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > Hein > > > > (via Tom White) > > > > > > > > > > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: [Shadow Regions / Read Replicas ] External replication disqualified?

2013-12-12 Thread Jonathan Hsieh
A little delayed but more questions. On Tue, Dec 3, 2013 at 10:41 PM, Devaraj Das wrote: > On Tue, Dec 3, 2013 at 6:49 PM, Jonathan Hsieh wrote: > > > The read replicas doc mentions something a little more intrusive in the > "3 > > options" section bu

Re: [Shadow Regions / Read Replicas ]

2013-12-12 Thread Jonathan Hsieh
a > > branch, and at the end of > > that, we are hoping we can have something useful and working (but without > > wal tailing and async > > wal replication), and we will have a more detailed plan for the remaining > > steps. We would love to > > hear more feedback of how to test / stabilize the feature at the merge > > discussions. > > > > Sounds good E. > > St.Ack > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: ANN: The third hbase-0.96.1 release candidate is available for download

2013-12-15 Thread Jonathan Hsieh
pachehbase-037/ > > > > 152 issues [1] have been fixed since 0.96.0, mostly performance > improvement > > and bug fixes. > > > > Shall we release this as hbase-0.96.1? Please vote by Monday, December > > 16th. > > > > Thanks, > > Your HBase Team > > > > 1. http://goo.gl/ZgYBW2 > > > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Rolling a quick patch release due to API incompatibility introduced in 0.96.1

2013-12-17 Thread Jonathan Hsieh
on it. (similar to 0.94.6.1). [0] https://issues.apache.org/jira/browse/HBASE-10186 [1] https://issues.apache.org/jira/browse/HBASE-10012 [2] https://issues.apache.org/jira/browse/HBASE-10188 Should be up for people to try shortly. Jon -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera

Re: Rolling a quick patch release due to API incompatibility introduced in 0.96.1

2013-12-17 Thread Jonathan Hsieh
anks, Jon. On Tue, Dec 17, 2013 at 12:08 PM, Andrew Purtell wrote: > Argh +1 > > > On Tue, Dec 17, 2013 at 8:41 AM, Jonathan Hsieh wrote: > > > Hey folks, > > > > I was reviewing some patches [0] and found that we introduced an API > > incompatibility in 0.

Re: Rolling a quick patch release due to API incompatibility introduced in 0.96.1

2013-12-17 Thread Jonathan Hsieh
Scratch that -- I was able to retrieve my passphrase. Currently uploading artifacts to my people.apache.org and to maven repo for release candidacy. Jon. On Tue, Dec 17, 2013 at 12:33 PM, Jonathan Hsieh wrote: > So I've gotten through most of the build/branch related processe

ANNOUNCE: hbase 0.96.1.1rc0 release candidate is available for download.

2013-12-17 Thread Jonathan Hsieh
k look and vote +1/-1 by 12/18/13 23:59 pacific time. If this passes we'll take down 0.96.1. Please check the release mechanics -- this is my first attempt at an hbase release. Thanks! [1] https://issues.apache.org/jira/browse/HBASE-10188 -- // Jonathan Hsieh (shay) // Software Engineer

Re: Please welcome our newest committer, Liang Xie!

2013-12-18 Thread Jonathan Hsieh
grats Liang! > > > >-Todd > > > > > >On Wed, Dec 18, 2013 at 4:53 PM, Ted Yu > > wrote: > > > >> Congratulations, Liang. > >> > >> On Dec 18, 2013, at 4:47 PM, Stack > > wrote: > >> > >> > 谢良 has been

Re: ANNOUNCE: hbase 0.96.1.1rc0 release candidate is available for download.

2013-12-19 Thread Jonathan Hsieh
in/hbase org.apache.hadoop.hbase.util.LoadTestTool -write > > 10:10:100 > > > > -num_keys 100 -read 100:30 > > > > -- bin/hbase > > "org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList > > > > Loop 1 1 300 /tmp/biglinkedlist 1&

Re: ANNOUNCE: hbase 0.96.1.1rc0 release candidate is available for download.

2013-12-19 Thread Jonathan Hsieh
that is fixed by the new lock?) Thanks, Jon. On Thu, Dec 19, 2013 at 9:01 AM, Ted Yu wrote: > Jon: > If Stack gives the greenlight, I can certainly port it to 0.96 branch. > > Cheers > > > On Thu, Dec 19, 2013 at 8:39 AM, Jonathan Hsieh wrote: > > > When I run the t

Re: ANNOUNCE: hbase 0.96.1.1rc0 release candidate is available for download.

2013-12-19 Thread Jonathan Hsieh
erver.hlog.lowreplication.rolllimit", 5); > > When consecutive log rolls exceeds the above conf value, lowReplicationRoll > is disabled. > > If replica count goes down because of issue in Datanodes, the above limit > would be reached ultimately. > > Cheers > > On Thu,

Re: ANNOUNCE: hbase 0.96.1.1rc0 release candidate is available for download.

2013-12-19 Thread Jonathan Hsieh
Vote passes with 5 binding +1's. ( apurtell, eclark, stack, enis, jmhsieh). Pushing out the release and announcement to user@. Thanks, Jon. On Thu, Dec 19, 2013 at 12:12 PM, Stack wrote: > On Thu, Dec 19, 2013 at 8:39 AM, Jonathan Hsieh wrote: > > > ... > > &

ANNOUNCE: hbase-0.96.1.1 available for download.

2013-12-19 Thread Jonathan Hsieh
ix [3] that addressed an api compatibility problem introduced between 0.96.0 ad 0.96.1. We will be removing 0.96.1. Thanks! Your HBase Team [1] http://www.apache.org/dyn/closer.cgi/hbase/ [2] http://hbase.apache.org/book.html#upgrade0.96 [3] http://goo.gl/q4S94X -- // Jonathan Hsieh (shay) // Sof

Re: ANNOUNCE: hbase-0.96.1.1 available for download.

2013-12-20 Thread Jonathan Hsieh
atch happened? Or is it only svn tag (and I am just assuming that works > > alright wrt not including 0.96.2 patches). > > > > > > On Thu, Dec 19, 2013 at 1:29 PM, Jonathan Hsieh > wrote: > > > > > The HBase Team is pleased to announce the immediate release of

Re: Re: About HBASE-3149

2013-12-23 Thread Jonathan Hsieh
gt; > Hi, > >Can anybody tell me if future HBase release will integrate 3149 for > > Make flush decisions per column family? > > > > By the way, for current HBase, if the simultaneous flush is the only > > issue? I mean, to create 10 CFs will not be

Re: Committing bug and test fixes to branches

2014-01-03 Thread Jonathan Hsieh
ting 0.96 back to a releasable state. > > Does trunk need a release manager? > > > Comments? > > -- Lars > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: Are we no longer giving credit in SVN commit messages?

2014-01-20 Thread Jonathan Hsieh
ch I'm not aware (please let me know!). > > -- > Best regards, > >- Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // j...@cloudera.com

Re: Are we no longer giving credit in SVN commit messages?

2014-01-20 Thread Jonathan Hsieh
$2}' | grep HBASE | head -1000 | grep -v \\: | grep -v \\. | wc 888 888 10011 On Mon, Jan 20, 2014 at 4:47 PM, Andrew Purtell wrote: > Oh, we don't want the "."? It's pretty universal in Hadoop land (just > checked common, hdfs, mapreduce, yarn, and

Re: 1.5.0-SNAPSHOT conflicting with hbase-proto 0.96.x

2014-01-27 Thread Jonathan Hsieh
method was changed, what's more with a > signature that renders it unusable. > > -- > Benoit "tsuna" Sigoure > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: Tagging 0.98.0 RC1 tomorrow

2014-01-29 Thread Jonathan Hsieh
tomorrow. > > -- > Best regards, > >- Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: Binary API compatibility is not a requirement for any 0.98 release candidate

2014-02-04 Thread Jonathan Hsieh
o RCs and I am hoping to get a third out tomorrow. > > -- > Best regards, > >- Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) > > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: Binary API compatibility is not a requirement for any 0.98 release candidate

2014-02-04 Thread Jonathan Hsieh
ompat not be a > criteria for 0.98 releases. Otherwise it would have been. :-) > > On Feb 4, 2014, at 2:46 PM, Jonathan Hsieh wrote: > > Andrew, > > I basically agree with lars here -- the ship has sailed here. However, > there are some patches that restored binary co

Re: DISCUSSION: 1.0.0

2014-02-11 Thread Jonathan Hsieh
C > > > >based > > > >actions and move everything else to plugins. > > > > > > > > > > > > > > > >-- > > > >Best regards, > > > > > > > > - Andy > > > > > > > >Problems worthy of attack prove their worth by hitting back. - Piet > Hein > > > >(via Tom White) > > > > > > > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: DISCUSSION: Enis for 1.0.0 RM

2014-02-13 Thread Jonathan Hsieh
d say let this be so. We could run a vote but my > > > > thinking is that any one interested in RM'ing can just volunteer to > do > > > the > > > > next one; it's healthy spreading the RM role IMO. > > > > > > > > St.Ack > > > > > > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: Public GitHub tree for HBASE-10191 work

2014-02-19 Thread Jonathan Hsieh
with > > poorly descriptive commit messages threaded throughout branch history. > > Maybe a weekly Sunday rebase (perhaps with squashing) and force push > would > > not be too onerous. > > > > -- > > Best regards, > > > >- Andy >

Re: Online merging of Regions

2014-03-01 Thread Jonathan Hsieh
he Regions. We then traverse > the final all merged leftist tree, write its data to a HFile and write the > new memstore being used for write's data into the HFile as well. > > It's just a thought.Please let me know your feedback and comments on it. > > Regards, > >

Re: Hadoop1 support in 0.98/1.0

2014-03-04 Thread Jonathan Hsieh
'll update that chart otherwise. > > > > While we are at it, should 1.0 support hadoop1 or not. I think it would > be > > good to keep support for h1 in 1.0, and drop it in 2.0 line. But it would > > mean if we do not drop support, we have to keep the support through 1.1, > > 1.2, etc. > > > > Enis > > > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: Publish 0.99.0-SNAPSHOT to Apache snapshots

2014-03-11 Thread Jonathan Hsieh
noon. > > > > -n > > > > [0]: http://hbase.apache.org/book.html#maven.snapshot > > > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: Please welcome our newest committer, Mr. Honghua Feng

2014-03-12 Thread Jonathan Hsieh
He has contributed some > > sweet > > > > patches since he started hanging out in these parts. Keep up the > great > > > > work Honghua. > > > > > > > > St.Ack > > > > > > > > > > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: [VOTE] The 3rd HBase 0.98.1 release candidate (RC2) is available for download

2014-03-27 Thread Jonathan Hsieh
new > blocker. > > > > Please try out the candidate and vote +1/-1 by midnight Pacific Time > (00:00 > > -0800 GMT) on March 31 on whether or not we should release this as > 0.98.1. > > > > -- > > Best regards, > > > >- Andy > > > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > > (via Tom White) > > > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: HBase region server failure issues

2014-04-15 Thread Jonathan Hsieh
edits. Jon. On Mon, Apr 14, 2014 at 10:37 PM, Vladimir Rodionov wrote: > Todd, how about 300 regions with 3x replication? Or 1000 regions? This is > going to be 3000 files. on HDFS. per one RS. When I said that it does not > scale, I meant that exactly that. > -- // Jon

Re: HBase region server failure issues

2014-04-15 Thread Jonathan Hsieh
t; > > On Tue, Apr 15, 2014 at 7:11 AM, Jonathan Hsieh wrote: > > > It makes sense to have as many wals as # of spindles / replication factor > > per machine. This should be decoupled from the number of regions on a > > region server. So for a cluster with 12 spindles

Re: HBase region server failure issues

2014-04-16 Thread Jonathan Hsieh
udiu Soroiu > > wrote: > > > > > > > > > > > > > After some tunning I managed to > > > > reduce it to 8 seconds in total and for the moment it fits the needs. > > > > > > > > > > What did you do Claudiu to get the time down? > > > Thanks, > > > St.Ack > > > > > > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: 0.99.0 and 1.0.0 targets in Jira

2014-04-16 Thread Jonathan Hsieh
rgets in Jira and some issues targeted to 0.99.0 and > some to 1.0.0. > > Which one should we use? > > -- Lars > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: 0.99.0 and 1.0.0 targets in Jira

2014-04-16 Thread Jonathan Hsieh
On Wed, Apr 16, 2014 at 9:40 PM, Nick Dimiduk wrote: > On Wed, Apr 16, 2014 at 9:28 PM, Jonathan Hsieh wrote: > > > I'd prefer if we got rid of one -- maybe mark all as 0.99 and remove > 1.0.0. > > When we branch 1.0, we rename 0.99 to 1.0 and create a 1.1-SNAPSHOT &

Re: Regarding removal of deprecated APIs

2014-04-23 Thread Jonathan Hsieh
> have seen . Do we have any guideline for removing those? (when) The > current Trunk going tp be 1.0, I think this is the time for cleanp.. What > do u say? > > -Anoop- > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: HBase failed in all connections

2014-04-30 Thread Jonathan Hsieh
the HBase Developer mailing list archive at Nabble.com. > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

[common type encoding breakout] Re: HBase Hackathon @ Salesforce 05/06/2014 notes

2014-05-12 Thread Jonathan Hsieh
types are slower than they should be and they start > coming up w/ their own encodings as workarounds; now you are back to > square one you need to version it all. > > Was proposed that we try and unify around a typing strategy that would work > as > the ONE way to do it in HBase with Phoenix going to try and come up on it > first. > To be continued up on the HBase dev list. > > Another proposal that had alot of nods in favor was that we not require > 1.0 and 2.0 to be compatible. > > Notes ran out at this stage. > > St.Ack > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: HMaster acts as HRegionServer when run on a single node

2014-05-14 Thread Jonathan Hsieh
king on HBASE-10336 "Remove > >> deprecated usage of Hadoop HttpServer in InfoServer", I first want to > >> assess the normal behavior before testing the changes. > >> > >> Thx, > >> > >> Eric > >> > >> [1] > >> > >> > https://raw.githubusercontent.com/datalayer/io.datalayer/master/hbase/server/src/main/resources/hbase-site.xml > >> > >> [2] > >> > >> > https://github.com/apache/hbase/commit/4430d10759f5e6392a0bd36395ba4f6bbc3bc94f > >> > > > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

hbase type encodings discussion part 2.

2014-05-16 Thread Jonathan Hsieh
8MURE/edit#heading=h.o1cgqtsqgqyg [3] https://docs.google.com/document/d/1BJooXphOduuPJHEfd3dAeF9Md3Zp2kRG31WExI4l88E/edit# [4] http://www.meetup.com/Hadoop-Summit-Community-San-Jose/events/179081342/?fromJoin=179081342 -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: [common type encoding breakout] Re: HBase Hackathon @ Salesforce 05/06/2014 notes

2014-05-19 Thread Jonathan Hsieh
>> > >>>>> Things get fuzzy around complex data types: pojos, compound rowkeys > (a > >>>>> special case of pojo), maps/dicts, and lists/arrays. These types are > >>>>> composed of other types and have different requirements based on &

Re: [VOTE] Merge branch HBASE-10070 to trunk

2014-06-06 Thread Jonathan Hsieh
t; > > https://issues.apache.org/jira/browse/HBASE-10792?jql=fixVersion%20%3D%20hbase-10070%20and%20fixversion%20%3D%200.99.0%20AND%20project%20%3D%20HBASE%20AND%20status%20%3D%20resolved > > [4] https://www.mail-archive.com/dev@hbase.apache.org/msg25795.html > > [5] > > > > > https://github.com/apache/hbase/commit/e22c7efeac02efde3451a0c9ff9bdcd2725576d0 > > [6] > > > > > http://www.slideshare.net/enissoz/hbase-high-availability-for-reads-with-time > > [7] https://issues.apache.org/jira/browse/HBASE-10070 > > > > > > -- > Best regards, > >- Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: [VOTE] Merge branch HBASE-10070 to trunk

2014-06-07 Thread Jonathan Hsieh
re on a > per-patch basis. > > It is possible. > Enis > > > On Fri, Jun 6, 2014 at 4:36 PM, Jonathan Hsieh wrote: > > > When we merged snapshots branch in we did this: > > > > t = trunk commit > > s = snapshot branch commit > > m = merge poin

Please fix your git name/address on your commits.

2014-06-10 Thread Jonathan Hsieh
Enis Soztutar fenghh Gary Helmling Jean-Daniel Cryans Jeffrey Zhong jeffreyz Jesse Yates Jimmy Xiang Jonathan Hsieh Jonathan M Hsieh jxiang larsh Lars Hofhansl liangxie Matteo Bertozzi mbertozzi Michael Stack Michael Stack ndimiduk Nick Dimiduk Nicolas Li

Re: Thinking of branching for 1.0 by June 23

2014-06-10 Thread Jonathan Hsieh
y alternate proposals? Which one we > should do? I am more in favor of b) which will be the most flexible route > to having true semantic versioning for the cost of added branch complexity. > a) should be fine as well if we are fine with lazy branching for 2.0. > > Enis > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: Anyone to have a quick look at 11313?

2014-06-11 Thread Jonathan Hsieh
te one? Unless I am not in the correct group or > > something... > > > > On Tue, Jun 10, 2014 at 07:58PM, Jean-Marc Spaggiari wrote: > > > Pretty trivial patch. > > > > > > Thanks, > > > > > > JM > > > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: [VOTE] Merge branch HBASE-10070 to trunk

2014-06-12 Thread Jonathan Hsieh
out painful, we'll revert to the > merge approach similar to what you describe. > > Enis > > > On Sat, Jun 7, 2014 at 7:41 AM, Jonathan Hsieh wrote: > > > On Fri, Jun 6, 2014 at 5:05 PM, Enis Söztutar > wrote: > > > > > My preference would be to

Re: DISCUSSION: 1.0.0

2014-07-07 Thread Jonathan Hsieh
t;>> log > > >>>>>>>> splitting (and Sergey S has submitted a patch for review), > another > > >>>> is > > >>>>>>>> abstraction of ZK watcher (this is what I've been working on in > > the > > >>&g

Re: what label is used for issues that are good on ramps for new contributors?

2014-08-05 Thread Jonathan Hsieh
oob" is used more regularly on HBase. > > I worry that "noob" might be viewed as pejorative and discourage some > potential contributors. > > [1]: http://hbase.apache.org/book.html#jira > [2]: http://s.apache.org/exr > [3]: http://s.apache.org/XS6 > > -- > Sean > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: what label is used for issues that are good on ramps for new contributors?

2014-08-05 Thread Jonathan Hsieh
re sense > > (or if something else would be better still)? > > > > -Sean > > > > > > On Tue, Aug 5, 2014 at 10:35 AM, Andrew Purtell > > wrote: > > > > > Yes we can change that. I volunteer to change it. What should the new > > label

Re: HConstants.ALL_VERSIONS

2014-08-06 Thread Jonathan Hsieh
ke > sure on the compaction side we take care of this in case we want to never > delete any version? > > JM > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: Incorrect docs in 0.98.3 and 0.98.4

2014-08-06 Thread Jonathan Hsieh
trunk, but for 0.98.3 and 0.98.4 the docs are the vanilla docs. > >> > >> cheers, > >> esteban. > >> > >> -- > >> Cloudera, Inc. > >> > > > > > > > > -- > > Best regards, > > > >- Andy > > &

Re: HBase shell compatibility needs

2014-08-14 Thread Jonathan Hsieh
ural sql-shell-style. I'd even consider a new "user shell" that is parallel to the current "hacker" shell, with the goal marginalizing the current shell for low level repair work. > [1]: https://groups.google.com/d/msg/jruby-users/qmLpZ7qDwZo/J_iYViplcq4J > > -- > Sean > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: May I ask why HBase choose to partition data by range?

2014-08-18 Thread Jonathan Hsieh
ses to partition data by range? Why not by > Hash or list? I belive in the design phase of HBase, this should be > discussed. > > Thanks in adavance. > > > > Nai Yan > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: HBase shell compatibility needs

2014-08-18 Thread Jonathan Hsieh
On Thu, Aug 14, 2014 at 1:44 PM, Sean Busbey wrote: > Responses inline > > > On Thu, Aug 14, 2014 at 3:16 PM, Jonathan Hsieh wrote: > > > I've always found working with the jruby part of the hbase shell jarring > > because many of the hbase commands don&

Re: [VOTE] Which Orca representation?

2014-08-21 Thread Jonathan Hsieh
> > > [x] JMS orca > >

[ANNOUNCE] New HBase PMC members: Matteo Bertozzi, Nick Dimiduk, and Jeffrey Zhong

2014-08-25 Thread Jonathan Hsieh
an active member of the HBase-related Apache Phoenix project. Please join me in congratulating Matteo, Nick and Jeffery on their new roles! Jon -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: thinking about supporting upgrades to HBase 1.x and 2.x

2014-08-29 Thread Jonathan Hsieh
; > Per suggestion of Sean in HBASE-11860 I'm sending this to the list to > > > discuss this idea: > > > > > > I've been thinking that we should support upgrades from HBase clusters > > > running 0.94 to HBase 1.x initially. Do you guys concur that we

Re: thinking about supporting upgrades to HBase 1.x and 2.x

2014-09-10 Thread Jonathan Hsieh
l upgrades, major & minor, could be done > "rolling" as proto bufs would ensure backward compatibility. > > This was a pretty important feature to allow us to upgrade live clusters > without down times. > --Suraj > > > > On Fri, Aug 29, 2014 at 3:16 PM, J

Anyone having problems committing code?

2014-09-17 Thread Jonathan Hsieh
Hey all, I was able to commit about 4 hours ago but the next patch I'm trying to commit has been failing with the no dav (bad password). and haven't seen any other commits since then. is this just me or is it affecting anyone else? Thanks, Jon. -- // Jonathan Hsieh (shay) // HBase

Re: Anyone having problems committing code?

2014-09-18 Thread Jonathan Hsieh
maven, > > it attempted to authenticate, and eventually it locked the account. I > > removed the offending stanza but I haven't tried since. > > > > -n > > > > On Wednesday, September 17, 2014, Jonathan Hsieh > wrote: > > > > > Hey all, > >

Re: HBCK need write permissions?

2014-12-02 Thread Jonathan Hsieh
re I open a JIRA. > > > > Why does HBCK need write permissions if there isn't any --repair ish flag > > specified? Should it not only be read permissions required? > > > > JM > > > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: Please welcome our latest committer, Sean Busbey

2014-12-04 Thread Jonathan Hsieh
Congrats Sean! On Thu, Dec 4, 2014 at 12:10 PM, Stack wrote: > Sean has been doing excellent work around these environs. Your PMC made him > a committer in recognition. Welcome Sean! > > St.Ack > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer

Re: ByteBuffer Backed Cell - New APIs (HBASE-12358)

2014-12-04 Thread Jonathan Hsieh
> When ever *hasArray() is false* - using getXXXArray() API would throw > *UnSupportedOperation > Exception*. > > As said if we want *getXXXArray()* to be supported as per the existing way > then getXXXBuffer() and getXXXOffset(), getXXXLength() should be clearly > documented. > > Thoughts!!! > > Regards > Ram & Anoop > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: ByteBuffer Backed Cell - New APIs (HBASE-12358)

2014-12-05 Thread Jonathan Hsieh
imilar in behaviour could lead to lot more confusion? > Internally it is more easy. > > > On Fri, Dec 5, 2014 at 11:10 AM, Jonathan Hsieh wrote: > > > Ram, > > > > Can we essentially do both by creating the new getXxBuffer, and then also > > creating new o

Re: Please welcome our latest new committer, Virag Kothari

2014-12-05 Thread Jonathan Hsieh
​ > . He's been a recent frequent contributor and collaborator on changes and > backports related to metadata tables, ZooKeeper, and the Master. > > -- > Best regards, > >- Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom

Re: Please welcome our latest committer, the mighty Liu Shaohui!

2015-01-07 Thread Jonathan Hsieh
the fold. > > Yours, > St.Ack > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh

Re: Review Request 4884: Backport HBASE-12533: staging directories are not deleted after secure bulk load

2015-02-04 Thread Jonathan Hsieh
adoop/hbase/mapreduce/TestLoadIncrementalHFiles.java <https://review.cloudera.org/r/4884/#comment12937> nit (fix upstream if there) - Jonathan Hsieh On Feb. 4, 2015, 9:55 p.m., Srikanth Srungarapu wrote: > > --- > This is

  1   2   3   4   5   6   7   8   >