Hi Chris and Andrew, The intend is for new code to have better unit test cases without resort to invocation of miniHDFSCluster or miniYarnCluster. Existing code don’t require refactoring, if the test cases already have good coverages. I am currently working on part of YARN to improve YARN and Docker integration. There are a lot of code getting triggered for UGI, FileSystem object to Yarn job submission. My code is only responsible to check the logic of the user input, and expected output prior to YarnClient job submission. Starting a miniCluster for this test case is excessive for the small piece of code for validation. The submission code was imported from Slider for YARN native services, a single class imports various Hadoop services. In several failure cases, it is difficult to simulate exact error conditions because the API is several layers deep. Powermock provides easy way to replace and stubbing return object or throw proper exception to simulate the failure conditions. One can argue that the code should have been written easier for unit tests, but Hadoop code density is beyond trivial to get simple initialization done. Constructor suppression, inner class replacement and private method override are good tools from Powermock that can provide more accurate testing without losing sights of multiple stage API calling tests while keeping the test case localized to a small piece of the greater puzzle. Hence, I like to request the community to rethink the improvement that Powermock can bring to the table. Thank you for your considation.
Regards, Eric From: Andrew Wang <andrew.w...@cloudera.com> Date: Friday, September 29, 2017 at 1:55 PM To: Chris Douglas <cdoug...@apache.org> Cc: Eric Yang <ey...@hortonworks.com>, "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org> Subject: Re: [DISCUSS] HADOOP-9122 Add power mock library for writing better unit tests Making code testable is about a lot more than your mocking library. In HDFS, the NameNode is very monolithic, so it's hard to instantiate little pieces of functionality in isolation and mock things out. I have my doubts that Powermock will help with this unless someone's willing to invest in significant refactoring effort of our existing code and unit test suites. I could see an argument of this being useful for new code being developed, but like Chris, I'd like to see an example of where Mockito falls short, and what additional capabilities Powermock brings to the table. Best, Andrew On Fri, Sep 29, 2017 at 10:38 AM, Chris Douglas <cdoug...@apache.org<mailto:cdoug...@apache.org>> wrote: Eric- Can you explain how Powermock differs from/augments Mockito, why we should adopt it, and maybe an example of an existing test that could be improved using this library? -C On Fri, Sep 29, 2017 at 10:12 AM, Eric Yang <ey...@hortonworks.com<mailto:ey...@hortonworks.com>> wrote: > Hi Hadoop-dev, > > Long time ago, Hadoop community decided to put Powermock on hold for unit > tests. Both mockito and powermock has evolved a lot in the past 5 years. > There are mature versions of both software, and there are compatibility > charts to indicate which versions can work together. Hadoop has grown a lot > in the last 5 years. It becomes apparent that without ability to instrument > lower level classes to contain unit test scope. Many tests are written to > simulate integration test in order to perform unit tests. The result is slow > performance on unit tests, and some parts are not testable strictly in unit > test case. This discussion is to revisit the decision, and see if we would > embrace Powermock and allow HADOOP-9122 to be implemented. Feel free to > comment on HADOOP-9122 and this thread to revisit this issue. > > Thank you for your time. > > Regards, > Eric --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org<mailto:common-dev-unsubscr...@hadoop.apache.org> For additional commands, e-mail: common-dev-h...@hadoop.apache.org<mailto:common-dev-h...@hadoop.apache.org>
