Re: Unit tests in hudi-client module fail due to SparkContext

Thanks for being so awesome, Raymond! On Tue, Jul 28, 2020 at 4:23 PM Shiyan Xu wrote: > yup i can make a PR for this. > > On Tue, Jul 28, 2020 at 2:30 PM Vinoth Chandar wrote: > > > Makes sense. Can we update some docs with this IDE setup? > > > > On Tue, Jul 28, 2020 at 10:32 AM Shiyan Xu >

Re: Unit tests in hudi-client module fail due to SparkContext

Makes sense. Can we update some docs with this IDE setup? On Tue, Jul 28, 2020 at 10:32 AM Shiyan Xu wrote: > Sure... here it is > https://gist.github.com/xushiyan/db4d4067657abe6b8872ef12473b7087 > > On Tue, Jul 28, 2020 at 9:53 AM Vinoth Chandar wrote: > > > Unfortunately, mailing list does

Re: Unit tests in hudi-client module fail due to SparkContext

Sure... here it is https://gist.github.com/xushiyan/db4d4067657abe6b8872ef12473b7087 On Tue, Jul 28, 2020 at 9:53 AM Vinoth Chandar wrote: > Unfortunately, mailing list does not support images . you could create a > gist and paste link :) > > On Tue, Jul 28, 2020 at 9:49 AM Y Ethan Guo >

Re: Unit tests in hudi-client module fail due to SparkContext

Unfortunately, mailing list does not support images . you could create a gist and paste link :) On Tue, Jul 28, 2020 at 9:49 AM Y Ethan Guo wrote: > Thanks, Shiyan. Interesting, maybe the memory config is the issue for me. > When I tried to run all tests, it fails. But running a smaller set

Re: Unit tests in hudi-client module fail due to SparkContext

Thanks, Shiyan. Interesting, maybe the memory config is the issue for me. When I tried to run all tests, it fails. But running a smaller set of tests per run is fine for me. I'll check that config in my settings. The discrepancy between the intellij test runner and CLI mvn runner may be >

Re: Unit tests in hudi-client module fail due to SparkContext

The maven surefire/failsafe plugin is configured with -Xmx2g here , which should be plentiful for all tests so far. The OOM looks weird to me.. maybe try checking the maven log see if -Xmx2g is indeed

Re: Unit tests in hudi-client module fail due to SparkContext

I see. I'll check the travis CI setup later. I'm unblocked now for running the unit tests locally. Thanks, - Ethan On Mon, Jul 27, 2020 at 11:03 PM Vinoth Chandar wrote: > So, i realized something. With the recent changes, functional tests retain > a single spark session for the entire test

Re: Unit tests in hudi-client module fail due to SparkContext

So, i realized something. With the recent changes, functional tests retain a single spark session for the entire test suite to speed things up. So thats probably what you were hitting first, when running via IDE Try following the .travis.yml profiles directly? Not sure about the OOM. Have not

Re: Unit tests in hudi-client module fail due to SparkContext

Hi Ethan, For purposes of unblocking yourself, can you try running them locally via mvn command via terminal? thanks vinoth On Sun, Jul 26, 2020 at 4:12 PM Y Ethan Guo wrote: > Hi, > > I'm working on hudi-client module and I notice that if I run all unit tests > under hudi-client locally in

Unit tests in hudi-client module fail due to SparkContext

Hi, I'm working on hudi-client module and I notice that if I run all unit tests under hudi-client locally in IntelliJ, some tests (54 out of 256) are failing due to the following SparkException: "Only one SparkContext may be running in this JVM". Is there any way I can get around this?