[jira] [Created] (HBASE-17574) Clean up how to run tests under hbase-spark module
Yi Liang created HBASE-17574:
Summary: Clean up how to run tests under hbase-spark module
Key: HBASE-17574
URL: https://issues.apache.org/jira/browse/HBASE-17574
Project: HBase
Issue Type: Bug
Components: spark
Affects Versions: 2.0.0
Reporter: Yi Liang
Assignee: Yi Liang
Fix For: 2.0.0
In master brunch, the test of hbase-spark module needs clean-up.
I think we need to let hbase-spark follow the rules that exist in the whole
hbase project
1. In hbase-spark, all the scala test cases are regarded as integration test,
i.e. we need to go to hbase-spark folder to use mvn verify to run the test
case. I think these tests had better to be regard as unit test for the
following reasons:
(1) All the scala test are very small, most of them can be finished within 20s.
(2) Integration test usually put into hbase-it module, not in its own module.
(3) Hadoop QA could not run those scala test in hbase-spark, I guess Hadoop QA
will only run mvn test under root dir, however hbase-spark need mvn verify.
(4) From its pom.xml below, you can see that, both
integration-test and test point to same
test. From MVN reference,
http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Built-in_Lifecycle_Bindings,
we know that if a goal is bound to one or more build phases, that goal will be
called in all those phases. it means that mvn test and mvn integration-test
will do same thing, however true in test phase just
disable the mvn test command. It is uncommon to have define like that.
{code}
test
test
test
true
integration-test
integration-test
test
Integration-Test
-Xmx1536m -XX:MaxPermSize=512m
-XX:ReservedCodeCacheSize=512m
false
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (HBASE-17573) Some refactor in proc wals code and adding comments on some corner cases.
Appy created HBASE-17573: Summary: Some refactor in proc wals code and adding comments on some corner cases. Key: HBASE-17573 URL: https://issues.apache.org/jira/browse/HBASE-17573 Project: HBase Issue Type: Task Reporter: Appy Previous comment which outlines the work to be done:https://issues.apache.org/jira/browse/HBASE-16524?focusedCommentId=15798436&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15798436 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (HBASE-17572) HMaster: Caught throwable while processing event C_M_MERGE_REGION (UndeclaredThrowableException)
Andrew Purtell created HBASE-17572:
--
Summary: HMaster: Caught throwable while processing event
C_M_MERGE_REGION (UndeclaredThrowableException)
Key: HBASE-17572
URL: https://issues.apache.org/jira/browse/HBASE-17572
Project: HBase
Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Andrew Purtell
Running ITBLL 1B rows against 1.3.0 compiled against Hadoop 2.7.3 with the
noKill monkey policy, I see both masters go down with
master.HMaster: Caught throwable while processing event C_M_MERGE_REGION
java.lang.reflect.UndeclaredThrowableException
In ServerManager#sendRegionsMerge we call ProtobufUtil#mergeRegions, which does
a doAs, and the code within that block invokes RSRpcServices#mergeRegions, but
is not resilient against RegionOpeningException ("region is opening")
An UndeclaredThrowableException is "thrown by a method invocation on a proxy
instance if its invocation handler's invoke method throws a checked exception
(a Throwable that is not assignable to RuntimeException or Error) that is not
assignable to any of the exception types declared in the throws clause of the
method that was invoked on the proxy instance and dispatched to the invocation
handler."
(http://docs.oracle.com/javase/7/docs/api/java/lang/reflect/UndeclaredThrowableException.html)
{noformat}
2017-01-31 07:21:17,495 FATAL [MASTER_TABLE_OPERATIONS-node-1:16000-0]
master.HMaster: Caught throwable while processing event C_M_MERGE_REGION
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1737)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.mergeRegions(ProtobufUtil.java:1990)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionsMerge(ServerManager.java:925)
at
org.apache.hadoop.hbase.master.handler.DispatchMergingRegionHandler.process(DispatchMergingRegionHandler.java:153)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.google.protobuf.ServiceException:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException):
org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7.
is opening on node-2.cluster,16020,1485822382322
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497)
at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22749)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2355)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:244)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:340)
at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.mergeRegions(AdminProtos.java:23695)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1993)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1990)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1719)
... 7 more
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException):
org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7.
is opening on node-2.cluster,16020,1485822382322
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497)
at
org.a
Re: Reminder: Please use git am when committing patches
Looks like replies are in favor of the idea that work done by our contributors should be credit appropriately and show up in places that matter. I'll make the changes in dev book to mention git am (if formatted patch) and git --author (if not formatted patch, alternatively ask the contributor to use the 'right' way and re-upload formatted patch). Here's the jira https://issues.apache.org/jira/browse/HBASE-17552. On Fri, Jan 27, 2017 at 1:08 PM, Sean Busbey wrote: > I'm a strong +1 for getting as many of our committers as possible over > to using git am or some way of making sure the contributor shows up as > the author. I agree with the previous statements about incentivizing > participation by making sure contributors get credit in places they'll > commonly look (e.g. github and other open source contribution > aggregators). Additionally, as a PMC member it is much easier to try > to build an idea of contributions to the repo over time if I can scan > through git author and sign-off tags than if I have to try to pick out > bespoke commit message details. > > > > On Thu, Jan 26, 2017 at 8:45 PM, Apekshit Sharma > wrote: > > Yes, many people who contribute to open source like their work to show up > > in git profile. I do for one. > > Past thread on this discussion: > > http://search-hadoop.com/m/HBase/YGbb14CnYDA6GX1/threaded > > > > On Thu, Jan 26, 2017 at 6:01 PM, 张铎(Duo Zhang) > > wrote: > > > >> I think the point here is github will not count the name in the commit > >> message as a contributor for the project? > >> > >> Enis Söztutar 于2017年1月27日 周五09:41写道： > >> > >> > Yep, we have been indeed using the msg (author name) historically. In > >> cases > >> > where author info is there, the guideline is to use git am --signoff. > >> > > >> > I don't think we should require git comit --author, as long as there > is > >> > attribution in the commit msg. But we can do a --author as an option. > >> > > >> > Enis > >> > > >> > On Thu, Jan 26, 2017 at 4:29 PM, 张铎(Duo Zhang) > > >> > wrote: > >> > > >> > > See here, our committer guide says that the commit message format > >> should > >> > be > >> > > HBASE-XXX XXX (the actual author). > >> > > > >> > > http://hbase.apache.org/book.html#_commit_message_format > >> > > > >> > > I think the rule was setup when we were still on svn, but we do not > >> > change > >> > > it. > >> > > > >> > > I‘m a big +1 on always using 'git am' if possible, and set > >> > > author explicitly when using 'git commit'. One more thing is that, > do > >> not > >> > > forget to add --signoff :) > >> > > > >> > > Mind opening a issue to modify the above section in hbase book, > Appy? > >> > > > >> > > Thanks. > >> > > > >> > > 2017-01-27 4:12 GMT+08:00 Chetan Khatri < > chetan.opensou...@gmail.com>: > >> > > > >> > > > Thanks , Appy. For understanding because i am very new to open > source > >> > > > contribution. > >> > > > > >> > > > On Fri, Jan 27, 2017 at 1:25 AM, Apekshit Sharma < > a...@cloudera.com> > >> > > > wrote: > >> > > > > >> > > > > Hi devs, > >> > > > > > >> > > > > A recent question by new contributor (thread: Git Pull Request > to > >> > > HBase) > >> > > > > and looking at history of commits, i think it is worth > re-iterating > >> > > that > >> > > > > committer should use *git am* while committing patches so that > >> > whoever > >> > > > did > >> > > > > the actual work gets appropriate credit in git history. > >> > > > > In case the patch uploaded is unformatted patch (without author > >> tag), > >> > > > > please use > >> > > > > *git commit --author=.* > >> > > > > I don't think we should be lax about it as a community if the > >> > original > >> > > > > author doesn't get appropriate credit. > >> > > > > > >> > > > > Thanks > >> > > > > -- Appy > >> > > > > > >> > > > > >> > > > >> > > >> > > > > > > > > -- > > > > -- Appy > -- -- Appy
Successful: HBase Generate Website
Build status: Successful
If successful, the website and docs have been generated. To update the live
site, follow the instructions below. If failed, skip to the bottom of this
email.
Use the following commands to download the patch and apply it to a clean branch
based on origin/asf-site. If you prefer to keep the hbase-site repo around
permanently, you can skip the clone step.
git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
cd hbase-site
wget -O-
https://builds.apache.org/job/hbase_generate_website/475/artifact/website.patch.zip
| funzip > b7fc7bf246934cea09e22f55bea62415f7319647.patch
git fetch
git checkout -b asf-site-b7fc7bf246934cea09e22f55bea62415f7319647
origin/asf-site
git am --whitespace=fix b7fc7bf246934cea09e22f55bea62415f7319647.patch
At this point, you can preview the changes by opening index.html or any of the
other HTML pages in your local
asf-site-b7fc7bf246934cea09e22f55bea62415f7319647 branch.
There are lots of spurious changes, such as timestamps and CSS styles in
tables, so a generic git diff is not very useful. To see a list of files that
have been added, deleted, renamed, changed type, or are otherwise interesting,
use the following command:
git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
To see only files that had 100 or more lines changed:
git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
When you are satisfied, publish your changes to origin/asf-site using these
commands:
git commit --allow-empty -m "Empty commit" # to work around a current ASF
INFRA bug
git push origin asf-site-b7fc7bf246934cea09e22f55bea62415f7319647:asf-site
git checkout asf-site
git branch -D asf-site-b7fc7bf246934cea09e22f55bea62415f7319647
Changes take a couple of minutes to be propagated. You can verify whether they
have been propagated by looking at the Last Published date at the bottom of
http://hbase.apache.org/. It should match the date in the index.html on the
asf-site branch in Git.
As a courtesy- reply-all to this email to let other committers know you pushed
the site.
If failed, see https://builds.apache.org/job/hbase_generate_website/475/console
