from:"Mingliang Liu"

Mingliang Liu created HDFS-15327:


 Summary: 
TestDataNodeErasureCodingMetrics.testReconstructionBytesPartialGroup3 fails 
intermittently
 Key: HDFS-15327
 URL: https://issues.apache.org/jira/browse/HDFS-15327
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, erasure-coding, test
Affects Versions: 3.4.0
Reporter: Mingliang Liu


Sample stack trace:

{code}
Error Message
ecReconstructionBytesRead should be  expected:<6501170> but was:<0>
Stacktrace
java.lang.AssertionError: ecReconstructionBytesRead should be  
expected:<6501170> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at 
org.apache.hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics.testReconstructionBytesPartialGroup3(TestDataNodeErasureCodingMetrics.java:150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}

Sample failing build:
- https://builds.apache.org/job/PreCommit-HDFS-Build/29226/testReport/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-15324) TestRefreshCallQueue::testRefresh fails intermittently



 [ 
https://issues.apache.org/jira/browse/HDFS-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu resolved HDFS-15324.
--
Resolution: Duplicate

> TestRefreshCallQueue::testRefresh fails intermittently
> --
>
> Key: HDFS-15324
> URL: https://issues.apache.org/jira/browse/HDFS-15324
> Project: Hadoop HDFS
>  Issue Type: Test
>  Components: namenode, test
>Affects Versions: 3.4.0
>        Reporter: Mingliang Liu
>Priority: Major
>
> This test seems flaky. It failed previously see HDFS-10253 and now it is 
> failing intermittently in {{trunk}}. Not sure we should use the same fix to 
> Mock as it was used in HDFS-10253.
> Sample failing builds:
> - 
> https://builds.apache.org/job/hadoop-multibranch/job/PR-1992/2/testReport/org.apache.hadoop/TestRefreshCallQueue/testRefresh/
> - 
> https://builds.apache.org/job/PreCommit-HDFS-Build/29221/testReport/org.apache.hadoop/TestRefreshCallQueue/testRefresh/
> Sample failing stack is:
> {code}
> Error Message
> org.apache.hadoop.TestRefreshCallQueue$MockCallQueue could not be constructed.
> Stacktrace
> java.lang.RuntimeException: 
> org.apache.hadoop.TestRefreshCallQueue$MockCallQueue could not be constructed.
>   at 
> org.apache.hadoop.ipc.CallQueueManager.createCallQueueInstance(CallQueueManager.java:193)
>   at 
> org.apache.hadoop.ipc.CallQueueManager.(CallQueueManager.java:83)
>   at org.apache.hadoop.ipc.Server.(Server.java:3087)
>   at org.apache.hadoop.ipc.RPC$Server.(RPC.java:1039)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:427)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:347)
>   at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:848)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:475)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:857)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:763)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:1014)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:987)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1756)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1332)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1101)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:974)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:906)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:534)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:493)
>   at 
> org.apache.hadoop.TestRefreshCallQueue.setUp(TestRefreshCallQueue.java:67)
>   at 
> org.apache.hadoop.TestRefreshCallQueue.testRefresh(TestRefreshCallQueue.java:115)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRun

[jira] [Created] (HDFS-15326) TestDataNodeErasureCodingMetrics::testFullBlock fails intermittently

Mingliang Liu created HDFS-15326:


 Summary: TestDataNodeErasureCodingMetrics::testFullBlock fails 
intermittently
 Key: HDFS-15326
 URL: https://issues.apache.org/jira/browse/HDFS-15326
 Project: Hadoop HDFS
  Issue Type: Test
  Components: datanode, test
Affects Versions: 3.4.0
Reporter: Mingliang Liu


Sample failing build: 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1992/2/testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeErasureCodingMetrics/testFullBlock/
Sample failing stack:
{code}
Error Message
Wrongly computed block reconstruction work
Stacktrace
java.lang.AssertionError: Wrongly computed block reconstruction work
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics.doTest(TestDataNodeErasureCodingMetrics.java:205)
at 
org.apache.hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics.testFullBlock(TestDataNodeErasureCodingMetrics.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-15324) TestRefreshCallQueue::testRefresh fails intermittently

Mingliang Liu created HDFS-15324:


 Summary: TestRefreshCallQueue::testRefresh fails intermittently
 Key: HDFS-15324
 URL: https://issues.apache.org/jira/browse/HDFS-15324
 Project: Hadoop HDFS
  Issue Type: Test
  Components: namenode, test
Affects Versions: 3.4.0
Reporter: Mingliang Liu


This test seems flaky. It failed previously see HDFS-10253 and now it is 
failing intermittently in {{trunk}}. Not sure we should use the same fix to 
Mock as it was used in HDFS-10253.

Sample failing builds:
- 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1992/2/testReport/org.apache.hadoop/TestRefreshCallQueue/testRefresh/
- 
https://builds.apache.org/job/PreCommit-HDFS-Build/29221/testReport/org.apache.hadoop/TestRefreshCallQueue/testRefresh/

Sample failing stack is:
{code}
Error Message
org.apache.hadoop.TestRefreshCallQueue$MockCallQueue could not be constructed.
Stacktrace
java.lang.RuntimeException: 
org.apache.hadoop.TestRefreshCallQueue$MockCallQueue could not be constructed.
at 
org.apache.hadoop.ipc.CallQueueManager.createCallQueueInstance(CallQueueManager.java:193)
at 
org.apache.hadoop.ipc.CallQueueManager.(CallQueueManager.java:83)
at org.apache.hadoop.ipc.Server.(Server.java:3087)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:1039)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:427)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:347)
at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:848)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:475)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:857)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:763)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:1014)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:987)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1756)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1332)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1101)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:974)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:906)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:534)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:493)
at 
org.apache.hadoop.TestRefreshCallQueue.setUp(TestRefreshCallQueue.java:67)
at 
org.apache.hadoop.TestRefreshCallQueue.testRefresh(TestRefreshCallQueue.java:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

2020-04-24 Thread Mingliang Liu

Brahma,

What about https://issues.apache.org/jira/browse/HADOOP-17007?

Thanks,

On Fri, Apr 24, 2020 at 11:07 AM Brahma Reddy Battula 
wrote:

> Ok. Done. Branch created.
>
> Following blockers are pending, will closely track this.
>
> https://issues.apache.org/jira/browse/HDFS-15287 ( Open: Under discussion
> )
> https://issues.apache.org/jira/browse/YARN-10194 ( Patch Available)
> https://issues.apache.org/jira/browse/HDFS-15286 ( Patch Available)
> https://issues.apache.org/jira/browse/YARN-9898 ( Patch Available)
>
>
> On Fri, Apr 24, 2020 at 7:42 PM Wei-Chiu Chuang
>  wrote:
>
> > +1 we should have the branch ASAP.
> >
> > On Wed, Apr 22, 2020 at 11:07 PM Akira Ajisaka 
> > wrote:
> >
> > > > Since blockers are not closed, I didn't cut the branch because
> > > multiple branches might confuse or sombody might miss to commit.
> > >
> > > The current situation is already confusing. The 3.3.1 version already
> > > exists in JIRA, so some committers wrongly commit non-critical issues
> to
> > > branch-3.3 and set the fix version to 3.3.1.
> > > I think now we should cut branch-3.3.0 and freeze source code except
> the
> > > blockers.
> > >
> > > -Akira
> > >
> > > On Tue, Apr 21, 2020 at 3:05 PM Brahma Reddy Battula <
> bra...@apache.org>
> > > wrote:
> > >
> > >> Sure, I will do that.
> > >>
> > >> Since blockers are not closed, I didn't cut the branch because
> > >> multiple branches might confuse or sombody might miss to commit.Shall
> I
> > >> wait till this weekend to create..?
> > >>
> > >> On Mon, Apr 20, 2020 at 11:57 AM Akira Ajisaka 
> > >> wrote:
> > >>
> > >>> Hi Brahma,
> > >>>
> > >>> Thank you for preparing the release.
> > >>> Could you cut branch-3.3.0? I would like to backport some fixes for
> > >>> 3.3.1 and not for 3.3.0.
> > >>>
> > >>> Thanks and regards,
> > >>> Akira
> > >>>
> > >>> On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula <
> > bra...@apache.org>
> > >>> wrote:
> > >>>
> >  Hi All,
> > 
> >  we are down to two blockers issues now (YARN-10194 and YARN-9848)
> > which
> >  are in patch available state.Hopefully we can out the RC soon.
> > 
> >  thanks to @Prabhu Joseph 
> > ,@masakate,@akira
> >  and @Wei-Chiu Chuang   and others for helping
> >  resloving the blockers.
> > 
> > 
> > 
> >  On Tue, Apr 14, 2020 at 10:49 PM Brahma Reddy Battula <
> >  bra...@apache.org> wrote:
> > 
> > >
> > > @Prabhu Joseph 
> > > >>> Have committed the YARN blocker YARN-10219 to trunk and
> > > cherry-picked to branch-3.3. Right now, there are two blocker
> Jiras -
> > > YARN-10233 and HADOOP-16982
> > > which i will help to review and commit. Thanks.
> > >
> > > Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982
> as
> > a
> > > blockers. (without YARN-10233 we have given so many releases,it's
> > not newly
> > > introduced.).. Thanks
> > >
> > > @Vinod Kumar Vavilapalli  ,@adam Antal,
> > >
> > > I noted YARN-9848 as a blocker as you mentioned above.
> > >
> > > @All,
> > >
> > > Currently following four blockers are pending for 3.3.0 RC.
> > >
> > > HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848.
> > >
> > >
> > >
> > > On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli <
> > > vino...@apache.org> wrote:
> > >
> > >> Looks like a really bad bug to me.
> > >>
> > >> +1 for revert and +1 for making that a 3.3.0 blocker. I think
> should
> > >> also revert it in a 3.2 maintenance release too.
> > >>
> > >> Thanks
> > >> +Vinod
> > >>
> > >> > On Apr 14, 2020, at 5:03 PM, Adam Antal <
> adam.an...@cloudera.com
> > .INVALID>
> > >> wrote:
> > >> >
> > >> > Hi everyone,
> > >> >
> > >> > Sorry for coming a bit late with this, but there's also one jira
> > >> that can
> > >> > have potential impact on clusters and we should talk about it.
> > >> >
> > >> > Steven Rand found this problem earlier and commented to
> > >> > https://issues.apache.org/jira/browse/YARN-4946.
> > >> > The bug has impact on the RM state store: the RM does not delete
> > >> apps - see
> > >> > more details in his comment here:
> > >> >
> > >>
> >
> https://issues.apache.org/jira/browse/YARN-4946?focusedCommentId=16898599=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16898599
> > >> > .
> > >> > (FYI He also created
> > >> https://issues.apache.org/jira/browse/YARN-9848 with
> > >> > the revert task).
> > >> >
> > >> > It might not be an actual blocker, but since there wasn't any
> > >> consensus
> > >> > about a follow up action, I thought we should decide how to
> > proceed
> > >> before
> > >> > release 3.3.0.
> > >> >
> > >> > Regards,
> > >> > Adam
> > >> >
> > >> > On Tue, Apr 14, 2020 at 9:35 AM Prabhu Joseph <
> > >>

[jira] [Created] (HDFS-15297) TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently in trunk

2020-04-23 Thread Mingliang Liu (Jira)

Mingliang Liu created HDFS-15297:


 Summary: TestNNHandlesBlockReportPerStorage::blockReport_02 fails 
intermittently in trunk
 Key: HDFS-15297
 URL: https://issues.apache.org/jira/browse/HDFS-15297
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: test
Affects Versions: 3.4.0
Reporter: Mingliang Liu


It fails intermittently on {{trunk}} branch. Not sure about other branches. 
Example builds are:
- 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1964/4/testReport/org.apache.hadoop.hdfs.server.datanode/TestNNHandlesBlockReportPerStorage/blockReport_02/
- 

Sample exception stack:
{quote}
java.lang.AssertionError: Wrong number of MissingBlocks is found expected:<2> 
but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at 
org.apache.hadoop.hdfs.server.datanode.BlockReportTestBase.blockReport_02(BlockReportTestBase.java:336)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-15296) TestBPOfferService::testMissBlocksWhenReregister is flaky

2020-04-23 Thread Mingliang Liu (Jira)

Mingliang Liu created HDFS-15296:


 Summary: TestBPOfferService::testMissBlocksWhenReregister is flaky
 Key: HDFS-15296
 URL: https://issues.apache.org/jira/browse/HDFS-15296
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: test
Affects Versions: 3.4.0
Reporter: Mingliang Liu


TestBPOfferService.testMissBlocksWhenReregister fails intermittently in 
{{trunk}} branch, not sure about other branches. Example failures are
- 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1964/4/testReport/org.apache.hadoop.hdfs.server.datanode/TestBPOfferService/testMissBlocksWhenReregister/
- 
https://builds.apache.org/job/PreCommit-HDFS-Build/29175/testReport/org.apache.hadoop.hdfs.server.datanode/TestBPOfferService/testMissBlocksWhenReregister/

Sample exception stack is:
{quote}
Stacktrace
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testMissBlocksWhenReregister(TestBPOfferService.java:350)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-14786) A new block placement policy tolerating availability zone failure

2019-08-27 Thread Mingliang Liu (Jira)

Mingliang Liu created HDFS-14786:

Summary: A new block placement policy tolerating availability zone
failure
Key: HDFS-14786
URL: https://issues.apache.org/jira/browse/HDFS-14786
Project: Hadoop HDFS
Issue Type: New Feature
Components: block placement
Reporter: Mingliang Liu

{{NetworkTopology}} assumes "/datacenter/rack/host" 3 layer topology. Default
block placement policies are rack awareness for better fault tolerance Newer
block placement policy like {{BlockPlacementPolicyRackFaultTolerant}} tries its
best to place the replicas to most racks, which further tolerates more racks
failing. [HADOOP-8470] brought {{NetworkTopologyWithNodeGroup}} to add another
layer under rack, i.e. "/datacenter/rack/host/nodegroup" 4 layer topology. With
that, replicas within a rack can be placed in different node groups for better
isolation.

Existing block placement policies tolerate rack failure since at least two
racks are chosen in those cases. Chances are all replicas could be placed in
the same datacenter, though there are multiple data centers in the same cluster
topology. In other words, fault of higher layers beyond rack is not well
tolerated.

However, more deployments in public cloud are leveraging multiple available
zones (AZ) for high-availability since the inter-AZ latency seems affordable in
many cases. In a single AZ, some cloud providers like AWS support [partitioned
placement
groups|https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html#placement-groups-partition]
which basically are different racks. A simple network topology mapped to HDFS
is "/availabilityzone/rack/host" 3 layers.

To achieve high availability tolerating zone failure, this JIRA proposes a new
data placement policy which tries its best to place replicas in most AZs, most
racks, and most evenly distributed.

Examples with 3 replicas, we choose racks as following:
# 1AZ: fall back to {{BlockPlacementPolicyRackFaultTolerant}} to most racks
# 2AZ: randomly choose one rack in 1st AZ and randomly choose two racks in the
other AZ
# 3AZ: randomly choose one rack in one AZ
# 4AZ: randomly choose three AZ and one rack in each AZ
After racks are chosen, hosts are chosen randomly honoring local storage,
favorite nodes, excluded nodes, storage types etc.

Data may become imbalance if topology is very uneven in AZs. This seems not a
problem as in public cloud, infrastructure provisioning is more flexible than
1P.

--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: HADOOP-14163 proposal for new hadoop.apache.org

2018-09-04 Thread Mingliang Liu

bout
> >>> the documentation which is generated by mvn-site (as before)
> >>>
> >>>
> >>> I got multiple valuable feedback and I improved the proposed site
> >>> according to the comments. Allen had some concerns about the used
> >>> technologies (hugo vs. mvn-site) and I answered all the questions why
> >>> I think mvn-site is the best for documentation and hugo is best for
> >>> generating site.
> >>>
> >>>
> >>> I would like to finish this effort/jira: I would like to start a
> >>> discussion about using this proposed version and approach as a new
> >>> site of Apache Hadoop. Please let me know what you think.
> >>>
> >>>
> >>> Thanks a lot,
> >>> Marton
> >>>
> >>> -
> >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

-- 
Mingliang Liu

Re: [VOTE] Release Apache Hadoop 2.8.4 (RC0)

2018-05-12 Thread Mingliang Liu

+1 (binding)

1. Download src file, and check md5
2. Build package from src
3. Create a 6 node cluster in Docker containers and start the HDFS/YARN
services (The script is caochong <https://github.com/weiqingy/caochong>).
4. Verify the running version
5. Run simple HDFS/YARN client/admin commands and verify the output
6. Run example programs wordcount and grep
7. Check NN/DN/RM status and logs

Thanks Junping for driving this!

On Tue, May 8, 2018 at 10:41 AM, 俊平堵 <junping...@apache.org> wrote:

> Hi all,
>  I've created the first release candidate (RC0) for Apache Hadoop
> 2.8.4. This is our next maint release to follow up 2.8.3. It includes 77
> important fixes and improvements.
>
> The RC artifacts are available at:
> http://home.apache.org/~junping_du/hadoop-2.8.4-RC0
>
> The RC tag in git is: release-2.8.4-RC0
>
> The maven artifacts are available via repository.apache.org<
> http://repository.apache.org> at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1118
>
> Please try the release and vote; the vote will run for the usual 5
> working days, ending on 5/14/2018 PST time.
>
> Thanks,
>
> Junping
>

-- 
Mingliang Liu

Re: [VOTE] Adopt HDSL as a new Hadoop subproject

2018-03-22 Thread Mingliang Liu

+1 (binding)

Thanks!

On Tue, Mar 20, 2018 at 11:20 AM, Owen O'Malley <owen.omal...@gmail.com>
wrote:

> All,
>
> Following our discussions on the previous thread (Merging branch HDFS-7240
> to trunk), I'd like to propose the following:
>
> * HDSL become a subproject of Hadoop.
> * HDSL will release separately from Hadoop. Hadoop releases will not
> contain HDSL and vice versa.
> * HDSL will get its own jira instance so that the release tags stay
> separate.
> * On trunk (as opposed to release branches) HDSL will be a separate module
> in Hadoop's source tree. This will enable the HDSL to work on their trunk
> and the Hadoop trunk without making releases for every change.
> * Hadoop's trunk will only build HDSL if a non-default profile is enabled.
> * When Hadoop creates a release branch, the RM will delete the HDSL module
> from the branch.
> * HDSL will have their own Yetus checks and won't cause failures in the
> Hadoop patch check.
>
> I think this accomplishes most of the goals of encouraging HDSL development
> while minimizing the potential for disruption of HDFS development.
>
> The vote will run the standard 7 days and requires a lazy 2/3 vote. PMC
> votes are binding, but everyone is encouraged to vote.
>
> +1 (binding)
>
> .. Owen
>



-- 
Mingliang Liu

Re: [DISCUSS] official docker image(s) for hadoop

2017-09-13 Thread Mingliang Liu

> It would be very helpful for testing the RC.
For testing and voting, I have been using docker containers for a while, see 
code at: https://github.com/weiqingy/caochong 


> TL;DR: I propose to create official hadoop images and upload them to the 
> dockerhub
I’m +1 on this idea. The “official” docker image basically means a commitment 
to maintain well documented and broadly tested images, which seems not a burden 
to us.

Ceph has a community docker project https://github.com/ceph/ceph-docker 
 and I think our scope here is similar to 
it.

Mingliang

> On Sep 13, 2017, at 11:39 AM, Yufei Gu  wrote:
> 
> It would be very helpful for testing the RC. To vote a RC, committers and
> PMCs usually spend lots of time to compile, deploy the RC, do several
> sanity tests, then +1 for the RC. The docker image potentially saves the
> compilation and deployment time, and people can do more tests.
> 
> Best,
> 
> Yufei
> 
> On Wed, Sep 13, 2017 at 11:19 AM, Wangda Tan  wrote:
> 
>> +1 to add Hadoop docker image for easier testing / prototyping, it gonna be
>> super helpful!
>> 
>> Thanks,
>> Wangda
>> 
>> On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
>> miklos.szeg...@cloudera.com> wrote:
>> 
>>> Marton, thank you for working on this. I think Official Docker images for
>>> Hadoop would be very useful for a lot of reasons. I think that it is
>> better
>>> to have a coordinated effort with production ready base images with
>>> dependent images for prototyping. Does anyone else have an opinion about
>>> this?
>>> 
>>> Thank you,
>>> Miklos
>>> 
>>> On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek  wrote:
>>> 
 
 TL;DR: I propose to create official hadoop images and upload them to
>> the
 dockerhub.
 
 GOAL/SCOPE: I would like improve the existing documentation with
 easy-to-use docker based recipes to start hadoop clusters with various
 configuration.
 
 The images also could be used to test experimental features. For
>> example
 ozone could be tested easily with these compose file and configuration:
 
 https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
 
 Or even the configuration could be included in the compose file:
 
 https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
 ker-compose.yaml
 
 I would like to create separated example compose files for federation,
>>> ha,
 metrics usage, etc. to make it easier to try out and understand the
 features.
 
 CONTEXT: There is an existing Jira https://issues.apache.org/jira
 /browse/HADOOP-13397
 But it’s about a tool to generate production quality docker images
 (multiple types, in a flexible way). If no objections, I will create a
 separated issue to create simplified docker images for rapid
>> prototyping
 and investigating new features. And register the branch to the
>> dockerhub
>>> to
 create the images automatically.
 
 MY BACKGROUND: I am working with docker based hadoop/spark clusters
>> quite
 a while and run them succesfully in different environments (kubernetes,
 docker-swarm, nomad-based scheduling, etc.) My work is available from
>>> here:
 https://github.com/flokkr but they could handle more complex use cases
 (eg. instrumenting java processes with btrace, or read/reload
>>> configuration
 from consul).
 And IMHO in the official hadoop documentation it’s better to suggest
>> to
 use official apache docker images and not external ones (which could be
 changed).
 
 Please let me know if you have any comments.
 
 Marton
 
 -
 To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
 For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
 
 
>>> 
>>

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

2017-09-10 Thread Mingliang Liu

Thanks Junping for doing this!

+1 (non-binding)

- Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
- Build package using maven (skipping tests) with Java 8
- Spin up a test cluster in Docker containers having 1 master node (NN/RM) and 
3 slave nodes (DN/NM)
- Operate the basic HDFS/YARN operations from command line, both client and 
admin
- Check NN/RM Web UI
- Run distcp to copy files from/to local and HDFS
- Run hadoop mapreduce examples: grep and wordcount
- Check the HDFS service logs

All looked good to me.

Mingliang

> On Sep 10, 2017, at 5:00 PM, Junping Du  wrote:
> 
> Hi folks,
> With fix of HADOOP-14842 get in, I've created our first release candidate 
> (RC0) for Apache Hadoop 2.8.2.
> 
> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
> will be the latest stable/production release for Apache Hadoop - it includes 
> 305 new fixed issues since 2.8.1 and 63 fixes are marked as blocker/critical 
> issues.
> 
>  More information about the 2.8.2 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> 
>  New RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
> 
>  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
> e6597fe3000b06847d2bf55f2bab81770f4b2505
> 
>  The maven artifacts are available via repository.apache.org at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1062
> 
>  Please try the release and vote; the vote will run for the usual 5 days, 
> ending on 09/15/2017 5pm PST time.
> 
> Thanks,
> 
> Junping
> 


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

2017-08-02 Thread Mingliang Liu

Thanks Konstantin!

+1 (non-binding)

I have tried several steps including:
- Download source package and check the checksums
- Build from source using Java 8
- Deploy a test cluster using docker containers w/ 5 DNs and 1 NN
- Run basic HDFS client/admin operations (e.g. read/write files, dfsadmin)
- Run NNThroughputBenchmark (using the new -fs option)
- Run DistCp and example apps (wordcount, sort)

All looked good to me.

> On Jul 29, 2017, at 4:29 PM, Konstantin Shvachko  wrote:
> 
> Hi everybody,
> 
> Here is the next release of Apache Hadoop 2.7 line. The previous stable
> release 2.7.3 was available since 25 August, 2016.
> Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
> critical bug fixes and major optimizations. See more details in Release
> Note:
> http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
> 
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
> 
> Please give it a try and vote on this thread. The vote will run for 5 days
> ending 08/04/2017.
> 
> Please note that my up to date public key are available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> Please don't forget to refresh the page if you've been there recently.
> There are other place on Apache sites, which may contain my outdated key.
> 
> Thanks,
> --Konstantin


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Corona is here -- And Ozone works !!!!

2017-07-20 Thread Mingliang Liu

Still, it’s good news!

Best,

> On Jul 20, 2017, at 2:44 PM, Anu Engineer  wrote:
> 
> Sorry, it was meant for a wrong alias. My apologies.
> 
> —Anu
> 
> 
> 
> 
> 
> On 7/20/17, 2:40 PM, "Anu Engineer"  wrote:
> 
>> Hi All,
>> 
>> I just deployed a test cluster with Nandakumar and we were able to run 
>> corona from a single node, with 10 thread for 12 mins.
>> 
>> We were able to write 789 MB and were writing 66 keys per second from a 
>> single node.
>> 
>> ***
>> Number of Volumes created: 10
>> Number of Buckets created: 10
>> Number of Keys added: 78984
>> Execution time: 12 minutes
>> 
>> 
>> The fun fact, Ozone just worked :), This is the first time we have written 
>> something like 78K keys into ozone. We are just starting on Corona and we 
>> will expand to run it from multiple nodes.
>> 
>> —Anu
>> 
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12064) Reuse object mapper in HDFS

2017-06-28 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-12064:


 Summary: Reuse object mapper in HDFS
 Key: HDFS-12064
 URL: https://issues.apache.org/jira/browse/HDFS-12064
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Mingliang Liu
Assignee: Hanisha Koneru
Priority: Minor


Currently there are a few places that are not following the recommended pattern 
of using object mapper - reuse if possible. Actually we can use 
{{ObjectReader}} or {{ObjectWriter}} to replace the object mapper in some 
places: they are straightforward and thread safe.

The benefit is all about performance, so in unit testing code I assume we don't 
have to worry too much.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11903) Cleaning up local storage when closing MiniOzoneCluster

2017-05-30 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-11903:


 Summary: Cleaning up local storage when closing MiniOzoneCluster
 Key: HDFS-11903
 URL: https://issues.apache.org/jira/browse/HDFS-11903
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Mingliang Liu
Assignee: Mingliang Liu


Currently we don't delete the local storage when closing {{MiniOzoneCluster}}, 
which will cause creating volume with the same name to fail, even in a new JVM 
process. Let's delete the local storage files: {{/tmp/ozone/metadata.db}} and 
{{/tmp/ozone/user.db}} when closing.

Please note this is not a complete solution. As the {{OzoneMetadataManager}}  
currently is a singleton, we are not able to create two independent 
MiniOzoneCluster in a single JVM. Data maybe shared unexpectedly. Let's address 
that in another JIRA, if needed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11892) Azure: handle failure gracefully in case of missing account key

2017-05-26 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-11892:


 Summary: Azure: handle failure gracefully in case of missing 
account key
 Key: HDFS-11892
 URL: https://issues.apache.org/jira/browse/HDFS-11892
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: fs/azure
Reporter: Mingliang Liu
Assignee: Mingliang Liu


Currently if the {{fs.azure.account.key.youraccount}} is missing, we will get 
error stack like this:

{code}
java.lang.IllegalArgumentException: The String is not a valid Base64-encoded 
string.

at com.microsoft.azure.storage.core.Base64.decode(Base64.java:63)
at 
com.microsoft.azure.storage.StorageCredentialsAccountAndKey.(StorageCredentialsAccountAndKey.java:81)
at 
org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.createStorageAccount(AzureBlobStorageTestAccount.java:464)
at 
org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.createTestAccount(AzureBlobStorageTestAccount.java:501)
at 
org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.create(AzureBlobStorageTestAccount.java:522)
at 
org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.create(AzureBlobStorageTestAccount.java:451)
at 
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization.createTestAccount(TestNativeAzureFileSystemAuthorization.java:50)
at 
org.apache.hadoop.fs.azure.AbstractWasbTestBase.setUp(AbstractWasbTestBase.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{code}
Actually, this error message is not very helpful. I'd admire you if you can 
immediately find the root cause of this failure.

If the test account is missing, we simply skip the test with meaningful error 
message. Here we should do the same thing: skip the test, and tell the user why 
we skip it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11704) OzoneFileSystem: A Hadoop file system implementation for Ozone

2017-04-25 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-11704:


 Summary: OzoneFileSystem: A Hadoop file system implementation for 
Ozone
 Key: HDFS-11704
 URL: https://issues.apache.org/jira/browse/HDFS-11704
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: fs/ozone
Reporter: Mingliang Liu
Assignee: Mingliang Liu






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-10891) Über-jira: Test Improvement by adding missing test cases for existing code

2016-12-28 Thread Mingliang Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-10891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu resolved HDFS-10891.
--
Resolution: Fixed

> Über-jira: Test Improvement by adding missing test cases for existing code
> --
>
> Key: HDFS-10891
> URL: https://issues.apache.org/jira/browse/HDFS-10891
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: test
>    Reporter: Mingliang Liu
>        Assignee: Mingliang Liu
>
> We have covered some test cases in our nightly run and we'd like to backport 
> those test cases to the community code branch. This work includes the efforts 
> but not limited to:
> # Add more *meaningful* test cases for existing code so it's hard to break it 
> as stability and compatibility should always be guarded if possible
> # Refactor existing tests that share common code snippet (e.g. helper 
> methods) and/or logic (e.g. set up a MiniDFSCluster). One approach that is 
> not popular (which it should) is @Parameterized tests
> # Reduce unnecessary overhead in unit tests (e.g. long interval sleep, build 
> MiniDFSCluster multiple times instead of reusing the same one). Some of the 
> cases were addressed by [HDFS-10666].
> This is not a long-term work to cover all future improvement in unit tests. 
> We'll be happy to resolve this after our internal test cases (will file 
> separate JIRAs for that) are mostly addressed/considered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-11105) TestRBWBlockInvalidation#testRWRInvalidation fails intermittently

2016-11-16 Thread Mingliang Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-11105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu reopened HDFS-11105:
--

> TestRBWBlockInvalidation#testRWRInvalidation fails intermittently
> -
>
> Key: HDFS-11105
> URL: https://issues.apache.org/jira/browse/HDFS-11105
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode, test
>Affects Versions: 2.8.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-11105.001.patch, HDFS-11105.002.patch
>
>
> The test {{TestRBWBlockInvalidation#testRWRInvalidation}}  fails 
> intermittently. The stack infos:
> {code}
> org.junit.ComparisonFailure: expected:<[old gs data
> new gs data
> ]> but was:<[]>
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation.testRWRInvalidation(TestRBWBlockInvalidation.java:225)
> {code}
> The issue is caused by the blocks reported not completed, similar to 
> HDFS-10499.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-11105) TestRBWBlockInvalidation#testRWRInvalidation fails intermittently

2016-11-16 Thread Mingliang Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-11105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu resolved HDFS-11105.
--
Resolution: Fixed

> TestRBWBlockInvalidation#testRWRInvalidation fails intermittently
> -
>
> Key: HDFS-11105
> URL: https://issues.apache.org/jira/browse/HDFS-11105
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode, test
>Affects Versions: 2.8.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-11105.001.patch, HDFS-11105.002.patch
>
>
> The test {{TestRBWBlockInvalidation#testRWRInvalidation}}  fails 
> intermittently. The stack infos:
> {code}
> org.junit.ComparisonFailure: expected:<[old gs data
> new gs data
> ]> but was:<[]>
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation.testRWRInvalidation(TestRBWBlockInvalidation.java:225)
> {code}
> The issue is caused by the blocks reported not completed, similar to 
> HDFS-10499.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-10666) Über-jira: Unit tests should not use fixed sleep interval to wait for conditions

2016-11-11 Thread Mingliang Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-10666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu resolved HDFS-10666.
--
  Resolution: Fixed
Target Version/s: 2.8.0  (was: 3.0.0-alpha2)

> Über-jira: Unit tests should not use fixed sleep interval to wait for 
> conditions
> 
>
> Key: HDFS-10666
> URL: https://issues.apache.org/jira/browse/HDFS-10666
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>        Reporter: Mingliang Liu
>    Assignee: Mingliang Liu
>
> There have been dozens of intermittent failing unit tests because they depend 
> on fixed-interval sleep to wait for conditions to reach before assertion. 
> This umbrella jira is to replace these sleep statements with:
> * {{GenericTestUtils.waitFor()}} to retry the conditions/assertions
> * Trigger internal state change of code to test, e.g. for {{MiniDFSCluster}} 
> we can trigger {BlockReports,HeartBeats,DeletionReports}
> * Fails fast if specific exceptions are caught
> * Coordinate the JUnit thread with activities of the internal threads
> * _ad-hoc fixes_...
> p.s. I don't know how closures in Java 8 comes into play but I'd like to see 
> any effort.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-11089) Accelerate TestDiskBalancerCommand using static shared MiniDFSCluster

2016-11-08 Thread Mingliang Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-11089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu resolved HDFS-11089.
--
Resolution: Not A Problem

Thanks [~arpitagarwal] for your suggestion. Closing this JIRA as "Not A 
Problem".

> Accelerate TestDiskBalancerCommand using static shared MiniDFSCluster
> -
>
> Key: HDFS-11089
> URL: https://issues.apache.org/jira/browse/HDFS-11089
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> It takes 50+ seconds to run the test suite. Similar to HDFS-11079, static 
> shared MiniDFSCluster will be used to accelerate the run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11095) BlockManagerSafeMode should respect extension period default config value (30s)

2016-11-02 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-11095:


 Summary: BlockManagerSafeMode should respect extension period 
default config value (30s)
 Key: HDFS-11095
 URL: https://issues.apache.org/jira/browse/HDFS-11095
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Reporter: Mingliang Liu
Assignee: Mingliang Liu


{code:title=BlockManagerSafeMode.java}
this.extension = conf.getInt(DFS_NAMENODE_SAFEMODE_EXTENSION_KEY, 0);
{code}
Though the default value (30s) is loaded from {{hdfs-default.xml}}, we should 
also respect this in the code by using 
{{DFSConfigKeys#DFS_NAMENODE_SAFEMODE_EXTENSION_DEFAULT}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11085) Add unit tests for NameNode failing to startup when name dir can not be written

2016-11-01 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-11085:


 Summary: Add unit tests for NameNode failing to startup when name 
dir can not be written
 Key: HDFS-11085
 URL: https://issues.apache.org/jira/browse/HDFS-11085
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode, test
Reporter: Mingliang Liu


This can be placed in {{org.apache.hadoop.hdfs.server.namenode.TestStartup}} 
test class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11083) Add unit test for DFSAdmin -report command

2016-11-01 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-11083:


 Summary: Add unit test for DFSAdmin -report command
 Key: HDFS-11083
 URL: https://issues.apache.org/jira/browse/HDFS-11083
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: shell, test
Reporter: Mingliang Liu


{{hdfs dfsadmin -report}} has very useful information about the cluster. There 
are some existing customized tools that depend on this command functionality. 
We should add unit test for it. Specially,
# If one datanode is dead, the report should indicate this
# If one block is corrupt, the "Missing blocks:" field should report this
# etc...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11031) Add additional unit test for DataNode startup behavior when volumes fail

2016-10-18 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-11031:


 Summary: Add additional unit test for DataNode startup behavior 
when volumes fail
 Key: HDFS-11031
 URL: https://issues.apache.org/jira/browse/HDFS-11031
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Mingliang Liu


There are several cases to add in {{TestDataNodeVolumeFailure}}:

- DataNode should not start in case of volumes failure
- DataNode should not start in case of lacking data dir read/write permission
- ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11030) TestDataNodeVolumeFailure#testVolumeFailure is flaky (though passing)

2016-10-18 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-11030:


 Summary: TestDataNodeVolumeFailure#testVolumeFailure is flaky 
(though passing)
 Key: HDFS-11030
 URL: https://issues.apache.org/jira/browse/HDFS-11030
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode, test
Affects Versions: 2.7.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu


TestDataNodeVolumeFailure#testVolumeFailure fails a volume and verifies the 
blocks and files are replicated correctly.

To fail a volume, it deletes all the blocks and sets the data dir read only.
{code}
// fail the volume
// delete/make non-writable one of the directories (failed volume)
data_fail = new File(dataDir, "data3");
failedDir = MiniDFSCluster.getFinalizedDir(dataDir, 
cluster.getNamesystem().getBlockPoolId());
if (failedDir.exists() &&
//!FileUtil.fullyDelete(failedDir)
!deteteBlocks(failedDir)
) {
  throw new IOException("Could not delete hdfs directory '" + failedDir + 
"'");
}
data_fail.setReadOnly();
failedDir.setReadOnly();
{code}
However, there are two bugs here:
- The {{failedDir}} directory for finalized blocks is not calculated correctly. 
It should use {{data_fail}} instead of {{dataDir}} as the base directory.
- When deleting block files in {{deteteBlocks(failedDir)}}, it assumes that 
there is no subdirectories in the data dir. This assumption was also noted in 
the comments.
{quote}
// we use only small number of blocks to avoid creating subdirs in the data 
dir..
{quote}
This is not true. On my local cluster and MiniDFSCluster, there will be 
subdir0/subdir0/ two level directories regardless of the number of blocks.

These two bugs made the blocks not deleted.

To fail a volume, it also needs to trigger the DataNode removing the volume and 
send block report to NN. This is basically in the {{triggerFailure()}} method.
{code}
  /**
   * go to each block on the 2nd DataNode until it fails...
   * @param path
   * @param size
   * @throws IOException
   */
  private void triggerFailure(String path, long size) throws IOException {
NamenodeProtocols nn = cluster.getNameNodeRpc();
List locatedBlocks =
  nn.getBlockLocations(path, 0, size).getLocatedBlocks();

for (LocatedBlock lb : locatedBlocks) {
  DatanodeInfo dinfo = lb.getLocations()[1];
  ExtendedBlock b = lb.getBlock();
  try {
accessBlock(dinfo, lb);
  } catch (IOException e) {
System.out.println("Failure triggered, on block: " + b.getBlockId() +  
"; corresponding volume should be removed by now");
break;
  }
}
  }
{code}
Accessing those blocks will not trigger failures if the directory is read-only 
(while the block files are all there). I ran the tests multiple times without 
triggering this failure. We have to write new block files to the data 
directories, or we should have deleted the blocks correctly.

This unit test has been there for years and it seldom fails, just because it's 
never triggered a real volume failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11008) Add unit test for parsing "-source" parameter in Balancer tool

2016-10-13 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-11008:


 Summary: Add unit test for parsing "-source" parameter in Balancer 
tool
 Key: HDFS-11008
 URL: https://issues.apache.org/jira/browse/HDFS-11008
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Mingliang Liu
Assignee: Mingliang Liu


[HDFS-8826] added an option to specify the source node list so that balancer 
only selects blocks to move from those nodes. This is very efficient in some 
cases.

The unit test of {{-source}} option parsing is missing, for both valid and 
invalid cases. This JIRA is to track the effort of adding unit test for that. 
In fact, we can refactor the existing code by replacing code that manually 
creates {{BalancerParameters}} with new code that explores the 
{{Balancer$Cli#parse()}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11007) AliyunOSSInputStream#read() should update read bytes stat correctly

2016-10-13 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-11007:


 Summary: AliyunOSSInputStream#read() should update read bytes stat 
correctly
 Key: HDFS-11007
 URL: https://issues.apache.org/jira/browse/HDFS-11007
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: tools
Reporter: Mingliang Liu
Assignee: Mingliang Liu


{code}
  @Override
  public synchronized int read() throws IOException {
..
if (statistics != null && byteRead >= 0) {
  statistics.incrementBytesRead(1);
}
return byteRead;
  }
{code}

I believe it should be {{statistics.incrementBytesRead(byteRead);}}?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11002) Fix broken attr(5) link in ExtendedAttributes.md

2016-10-12 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-11002:


 Summary: Fix broken attr(5) link in ExtendedAttributes.md
 Key: HDFS-11002
 URL: https://issues.apache.org/jira/browse/HDFS-11002
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.7.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu


http://www.bestbits.at/acl/man/man5/attr.txt is broken though the website is 
not. I suggest we use the http://man7.org/linux/man-pages/man5/attr.5.html 
which is mainted by maintainer of the Linux 
[man-pages|https://www.kernel.org/doc/man-pages/] project.

The same to http://www.bestbits.at/acl/man/man1/getfattr.txt => 
http://man7.org/linux/man-pages/man1/getfattr.1.html
and http://www.bestbits.at/acl/man/man1/setfattr.txt => 
http://man7.org/linux/man-pages/man1/setfattr.1.html

[~ajisakaa], thoughts? Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10986) DFSAdmin should show detailed error message if any

2016-10-07 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-10986:


 Summary: DFSAdmin should show detailed error message if any
 Key: HDFS-10986
 URL: https://issues.apache.org/jira/browse/HDFS-10986
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: tools
Affects Versions: 2.8.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu


There are some subcommands in {{DFSAdmin}} that swallow IOException and give 
very limited error message, if any, to the stderr.

{code}
$ hdfs dfsadmin -getDatanodeInfo localhost:9866
Datanode unreachable.
$ hdfs dfsadmin -evictWriters 127.0.0.1:9866
$ echo $?
-1
{code}

We should fix this by providing detailed error message. Actually, the 
{{DFSAdmin#run}} already handles exception carefully. All we need to do is to 
not swallow exceptions without good reason.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10985) o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs fails intermittently

2016-10-07 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-10985:


 Summary: 
o.a.h.ha.TestZKFailoverController#testGracefulFailoverMultipleZKfcs fails 
intermittently
 Key: HDFS-10985
 URL: https://issues.apache.org/jira/browse/HDFS-10985
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ha
Reporter: Mingliang Liu
Assignee: Mingliang Liu


h6. Error Message
{quote}
Unable to become active. Local node did not get an opportunity to do so from 
ZooKeeper, or the local node took too long to transition to active.
{quote}
h6. Stacktrace
{quote}
org.apache.hadoop.ha.ServiceFailedException: Unable to become active. Local 
node did not get an opportunity to do so from ZooKeeper, or the local node took 
too long to transition to active.
at 
org.apache.hadoop.ha.ZKFailoverController.doGracefulFailover(ZKFailoverController.java:690)
at 
org.apache.hadoop.ha.ZKFailoverController.access$400(ZKFailoverController.java:62)
at 
org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:607)
at 
org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:604)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1795)
at 
org.apache.hadoop.ha.ZKFailoverController.gracefulFailoverToYou(ZKFailoverController.java:604)
at 
org.apache.hadoop.ha.ZKFCRpcServer.gracefulFailover(ZKFCRpcServer.java:94)
at 
org.apache.hadoop.ha.TestZKFailoverController.testGracefulFailoverMultipleZKfcs(TestZKFailoverController.java:590)
{quote}

See recent failing build:
# 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10705/testReport/org.apache.hadoop.ha/TestZKFailoverController/testGracefulFailoverMultipleZKfcs/
# to add more

I think we can use {{GenericTestUtils.waitFor()}} to retry the assertions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10893) Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test

2016-09-22 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-10893:


 Summary: Refactor TestDFSShell by setting up MiniDFSCluser once 
for all commands test
 Key: HDFS-10893
 URL: https://issues.apache.org/jira/browse/HDFS-10893
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Priority: Minor


This is a minor change. It seems that setting up MiniDFSCluser once for all 
commands test will reduce the total time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10892) Add unit tests for HDFS command 'dfs -tail'

2016-09-22 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-10892:


 Summary: Add unit tests for HDFS command 'dfs -tail'
 Key: HDFS-10892
 URL: https://issues.apache.org/jira/browse/HDFS-10892
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs
Reporter: Mingliang Liu
Assignee: Mingliang Liu


I did not find unit test for HDFS command 'dfs -tail' in {{trunk}} code. I 
think it still merits to have one though the command itself has been used there 
for years.

It also happens to 'dfs -test'. Perhaps we can consolidate the patches here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10891) Über-jira: Test Improvement by adding missing test cases for existing code

2016-09-22 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-10891:


 Summary: Über-jira: Test Improvement by adding missing test cases 
for existing code
 Key: HDFS-10891
 URL: https://issues.apache.org/jira/browse/HDFS-10891
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: test
Reporter: Mingliang Liu
Assignee: Mingliang Liu


We have covered some test cases in our nightly run and we'd like to backport 
those test cases to the community code branch. This work includes the efforts 
but not limited to:
# Add more *meaningful* test cases for existing code so it's hard to break it 
as stability and compatibility should always be guarded if possible
# Refactor existing tests that share common code snippet (e.g. helper methods) 
and/or logic (e.g. set up a MiniDFSCluster). One approach that is not popular 
(which it should) is @Parameterized tests
# Reduce unnecessary overhead in unit tests (e.g. long interval sleep, build 
MiniDFSCluster multiple times instead of reusing the same one)

This is not a long-term work to cover all future improvement in unit tests. 
We'll be happy to resolve this after our internal test cases (will file 
separate JIRAs for that) are mostly addressed/considered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10807) Doc about upgrading to a version of HDFS with snapshots may be confusing

2016-08-26 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-10807:


 Summary: Doc about upgrading to a version of HDFS with snapshots 
may be confusing
 Key: HDFS-10807
 URL: https://issues.apache.org/jira/browse/HDFS-10807
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: documentation
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Priority: Minor


{code}
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsSnapshots.md 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsSnapshots.md
index 94a37cd..d856e8c 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsSnapshots.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsSnapshots.md
@@ -113,7 +113,7 @@ Upgrading to a version of HDFS with snapshots
 
 The HDFS snapshot feature introduces a new reserved path name used to
 interact with snapshots: `.snapshot`. When upgrading from an
-older version of HDFS, existing paths named `.snapshot` need
+older version of HDFS which does not support snapshots, existing paths named 
`.snapshot` need
 to first be renamed or deleted to avoid conflicting with the reserved path.
 See the upgrade section in
 [the HDFS user guide](HdfsUserGuide.html#Upgrade_and_Rollback)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-10677) Über-jira: Enhancements to NNThroughputBenchmark tool

2016-08-15 Thread Mingliang Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-10677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu resolved HDFS-10677.
--
   Resolution: Fixed
Fix Version/s: 2.8.0

> Über-jira: Enhancements to NNThroughputBenchmark tool
> -
>
> Key: HDFS-10677
> URL: https://issues.apache.org/jira/browse/HDFS-10677
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: benchmarks, tools
>    Reporter: Mingliang Liu
>        Assignee: Mingliang Liu
> Fix For: 2.8.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10747) o.a.h.hdfs.tools.DebugAdmin usage message is misleading

2016-08-10 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-10747:


 Summary: o.a.h.hdfs.tools.DebugAdmin usage message is misleading
 Key: HDFS-10747
 URL: https://issues.apache.org/jira/browse/HDFS-10747
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs-client
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Priority: Minor


[HDFS-6917] added a helpful hdfs debug command to validate blocks and call 
recoverlease. The usage doc is kinda misleading, as following:

{code}
$ hdfs debug verify
creating a new configuration
verify [-meta ] [-block ]
  Verify HDFS metadata and block files.  If a block file is specified, we
  will verify that the checksums in the metadata file match the block
  file.
{code}
Actually the {{-meta }} is necessary. {{[]}} is for optional 
arguments, if we follow the 
[convention|http://pubs.opengroup.org/onlinepubs/9699919799].

{code}
$ hdfs debug recoverLease
creating a new configuration
recoverLease [-path ] [-retries ]
  Recover the lease on the specified path.  The path must reside on an
  HDFS filesystem.  The default number of retries is 1.
{code}
{{-path }} is also the same case.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10726) org.apache.hadoop.http.TestHttpServerLifecycle times out

2016-08-05 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-10726:


 Summary: org.apache.hadoop.http.TestHttpServerLifecycle times out
 Key: HDFS-10726
 URL: https://issues.apache.org/jira/browse/HDFS-10726
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0-alpha2
Reporter: Mingliang Liu


{{org.apache.hadoop.http.TestHttpServerLifecycle}} times out in a few of r 
recent builds, e.g.
# 
https://issues.apache.org/jira/browse/HADOOP-13470?focusedCommentId=15408969=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15408969
# 
https://issues.apache.org/jira/browse/HADOOP-13466?focusedCommentId=15408342=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15408342
# 
https://issues.apache.org/jira/browse/HADOOP-13444?focusedCommentId=15400162=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15400162

Can not reproduced locally so the test failure is intermittent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10725) Caller context should always be constructed by a builder

2016-08-05 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-10725:


 Summary: Caller context should always be constructed by a builder
 Key: HDFS-10725
 URL: https://issues.apache.org/jira/browse/HDFS-10725
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ipc
Affects Versions: 2.8.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Priority: Trivial


Currently {{CallerContext}} is constructed by a builder. In this pattern, the 
constructor should be private so that caller context should always be 
constructed by a builder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10724) Document the caller context config keys

2016-08-04 Thread Mingliang Liu (JIRA)

Mingliang Liu created HDFS-10724:


 Summary: Document the caller context config keys
 Key: HDFS-10724
 URL: https://issues.apache.org/jira/browse/HDFS-10724
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ipc, namenode
Affects Versions: 2.8.0
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Priority: Minor


The three config keys {{hadoop.caller.context.enabled}}, 
{{hadoop.caller.context.max.size}} and 
{{hadoop.caller.context.signature.max.size}} are not documented in the 
{{core-default.xml}} file. This jira is to address this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

2016-07-25 Thread Mingliang Liu

Thanks Vinod.

+1 (non-binding)

* Downloaded and built from source (with Java 8)
* Checked LICENSE and NOTICE files
* Verified MD5 signatures
* Installed pseudo-distributed instance (Mac OS X 10.11)
* Ran through HDFS and mapreduce tests
* Operate HDFS from command line
* Run tools like distcp/NNThroughputBenchmark

L

> On Jul 22, 2016, at 7:15 PM, Vinod Kumar Vavilapalli  
> wrote:
> 
> Hi all,
> 
> I've created a release candidate RC0 for Apache Hadoop 2.7.3.
> 
> As discussed before, this is the next maintenance release to follow up 2.7.2.
> 
> The RC is available for validation at: 
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
> 
> 
> The RC tag in git is: release-2.7.3-RC0
> 
> The maven artifacts are available via repository.apache.org 
>  at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
> 
> 
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html 
>  for 
> your quick perusal.
> 
> As you may have noted, a very long fix-cycle for the License & Notice issues 
> (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip 
> by quite a bit. This release's related discussion thread is linked below: [1].
> 
> Please try the release and vote; the vote will run for the usual 5 days.
> 
> Thanks,
> Vinod
> 
> [1]: 2.7.3 release plan: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
> 


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10678) Documenting NNThroughputBenchmark tool

Mingliang Liu created HDFS-10678:


 Summary: Documenting NNThroughputBenchmark tool
 Key: HDFS-10678
 URL: https://issues.apache.org/jira/browse/HDFS-10678
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Mingliang Liu
Assignee: Mingliang Liu


The best (only) documentation for the NNThroughputBenchmark currently exists as 
a JavaDoc on the NNThroughputBenchmark class. This is less than useful, 
especially since we no longer generate javadocs for HDFS as part of the build 
process. I suggest we extract it into a separate markdown doc, or merge it with 
other benchmarking materials (if any?) about HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10677) Enhancements to NNThroughputBenchmark

Mingliang Liu created HDFS-10677:


 Summary: Enhancements to NNThroughputBenchmark
 Key: HDFS-10677
 URL: https://issues.apache.org/jira/browse/HDFS-10677
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: benchmarks
Reporter: Mingliang Liu
Assignee: Mingliang Liu






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10668) Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount

Mingliang Liu created HDFS-10668:


 Summary: Fix intermittently failing UT 
TestDataNodeMXBean#testDataNodeMXBeanBlockCount
 Key: HDFS-10668
 URL: https://issues.apache.org/jira/browse/HDFS-10668
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Mingliang Liu
Assignee: Mingliang Liu


h6.Error Message
{code}
After delete one file expected:<4> but was:<5>
{code}

h6. Stacktrace
{code}
java.lang.AssertionError: After delete one file expected:<4> but was:<5>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at 
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMXBean.testDataNodeMXBeanBlockCount(TestDataNodeMXBean.java:124)
{code}

Sample failing Jenkins pre-commit built, see 
[here|https://builds.apache.org/job/PreCommit-HDFS-Build/16094/testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeMXBean/testDataNodeMXBeanBlockCount/].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10666) Über-jira: Unit tests should not depend on nondeterministic behavior using fixed sleep interval

Mingliang Liu created HDFS-10666:


 Summary: Über-jira: Unit tests should not depend on 
nondeterministic behavior using fixed sleep interval
 Key: HDFS-10666
 URL: https://issues.apache.org/jira/browse/HDFS-10666
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 3.0.0-alpha2
Reporter: Mingliang Liu
Assignee: Mingliang Liu


There have been dozens of intermittent failing unit tests because they depend 
on fixed-interval sleep to wait for conditions to reach before assertion. This 
umbrella jira is to replace these sleep statements with:
* {{GenericTestUtils.waitFor()}} to retry the conditions/assertions
* Trigger internal state change of {{MiniDFSCluster}}, e.g. 
{{trigger\{BlockReports,HeartBeats,DeletionReports\}}}
* fails fast if specific exceptions are caught
* _ad-hoc fixes_ (TBD)

p.s. I don't know how closures in Java 8 comes into play but I'd like to see 
any effort.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10644) Fix intermittently failing TestDFSUpgradeWithHA#testRollbackWithJournalNodes

Mingliang Liu created HDFS-10644:


 Summary: Fix intermittently failing 
TestDFSUpgradeWithHA#testRollbackWithJournalNodes
 Key: HDFS-10644
 URL: https://issues.apache.org/jira/browse/HDFS-10644
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 3.0.0-alpha2
Reporter: Mingliang Liu


See [example stack trace | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/9996/testReport/org.apache.hadoop.hdfs.server.namenode.ha/TestDFSUpgradeWithHA/testRollbackWithJournalNodes/]
 along with stand output/error.

h6. Stacktrace
{code}
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA.testRollbackWithJournalNodes(TestDFSUpgradeWithHA.java:687)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10642) Fix intermittently failing UT TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas

Mingliang Liu created HDFS-10642:


 Summary: Fix intermittently failing UT 
TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas
 Key: HDFS-10642
 URL: https://issues.apache.org/jira/browse/HDFS-10642
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, test
Affects Versions: 3.0.0-alpha2
Reporter: Mingliang Liu


See [example stack trace | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/10001/testReport/org.apache.hadoop.hdfs.server.datanode.fsdataset.impl/TestLazyPersistReplicaRecovery/testDnRestartWithSavedReplicas/].

h6. Error Message
{code}
 Expected: is 
 but: was 
{code}
h6. Stacktrace
{code}
java.lang.AssertionError: 
Expected: is 
 but: was 
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:865)
at org.junit.Assert.assertThat(Assert.java:832)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.ensureFileReplicasOnStorageType(LazyPersistTestCase.java:141)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery.testDnRestartWithSavedReplicas(TestLazyPersistReplicaRecovery.java:53)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10641) Fix intermittently failing TestBlockManager#testBlockReportQueueing

Mingliang Liu created HDFS-10641:


 Summary: Fix intermittently failing 
TestBlockManager#testBlockReportQueueing
 Key: HDFS-10641
 URL: https://issues.apache.org/jira/browse/HDFS-10641
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs, test
Reporter: Mingliang Liu


See example failing [stack 
trace|https://builds.apache.org/job/PreCommit-HADOOP-Build/9996/testReport/org.apache.hadoop.hdfs.server.blockmanagement/TestBlockManager/testBlockReportQueueing/].

h6. Stacktrace
{code}
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager.testBlockReportQueueing(TestBlockManager.java:1074)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-9277) IOException "Unable to load OAuth2 connection factory." in TestWebHDFSOAuth2.listStatusReturnsAsExpected