Re: hudi dependency conflicts for test

Thanks Lian! Will work it in! On Tue, May 26, 2020 at 9:02 AM Lian Jiang wrote: > I added a comment in this wiki. Hope this works. Thanks. > > On Sun, May 24, 2020 at 2:32 AM Vinoth Chandar wrote: > > > Great team work everyone! > > > > Anything worth documenting here? > >

Re: hudi dependency conflicts for test

I added a comment in this wiki. Hope this works. Thanks. On Sun, May 24, 2020 at 2:32 AM Vinoth Chandar wrote: > Great team work everyone! > > Anything worth documenting here? > https://cwiki.apache.org/confluence/display/HUDI/Troubleshooting+Guide > > On Thu, May 21, 2020 at 11:02 PM Lian

Re: hudi dependency conflicts for test

Great team work everyone! Anything worth documenting here? https://cwiki.apache.org/confluence/display/HUDI/Troubleshooting+Guide On Thu, May 21, 2020 at 11:02 PM Lian Jiang wrote: > The root cause is that I need to use java 8 instead of the default java 11 > in intellij. Thanks everyone for

Re: hudi dependency conflicts for test

The root cause is that I need to use java 8 instead of the default java 11 in intellij. Thanks everyone for helping and cheers! On Thu, May 21, 2020 at 1:09 PM Lian Jiang wrote: > The examples in quick start work for me in spark-shell. I am trying to use > scala unit test to make these examples

Re: hudi dependency conflicts for test

The examples in quick start work for me in spark-shell. I am trying to use scala unit test to make these examples easier to repeat in CICD given hudi is still in incubating. Below is the new set of dependencies as instructed: compile group: 'org.apache.spark', name: 'spark-core_2.11', version:

Re: hudi dependency conflicts for test

hello jiang, Please try following demo, need spark(>=2.4.4) -- export SPARK_HOME=/work/BigData/install/spark/spark-2.4.5-bin-hadoop2.7 ${SPARK_HOME}/bin/spark-shell \ --packages

Re: hudi dependency conflicts for test

That was a close one. :) On Thu, May 21, 2020 at 10:46 AM Vinoth Chandar wrote: > Wow.. Race condition :) .. > > Thanks for racing , Raymond! > > On Thu, May 21, 2020 at 10:08 AM Shiyan Xu > wrote: > > > Hi Lian, it appears that you need to have spark-avro_2.11:2.4.4 in your > > classpath. > >

Re: hudi dependency conflicts for test

Hi Lian, >From your 2nd email seems like you downgrade spark to 2.3. Could you try using spark 2.4+ Please also refer to the release note for dependency versions. https://hudi.apache.org/releases.html#release-051-incubating-docs On Thu, May 21, 2020 at 11:59 AM Lian Jiang wrote: > Thanks

Re: hudi dependency conflicts for test

Thanks Shiyan and Vinoth. Unfortunately, adding org.apache.spark:spark-avro_2.11:2.4.4 throws another version related exception: java.lang.NoSuchMethodError: org.apache.avro.Schema.createUnion([Lorg/apache/avro/Schema;)Lorg/apache/avro/Schema; at

Re: hudi dependency conflicts for test

Wow.. Race condition :) .. Thanks for racing , Raymond! On Thu, May 21, 2020 at 10:08 AM Shiyan Xu wrote: > Hi Lian, it appears that you need to have spark-avro_2.11:2.4.4 in your > classpath. > > > > On Thu, May 21, 2020 at 10:04 AM Lian Jiang wrote: > > > Thanks Balaji. > > > > My unit test

Re: hudi dependency conflicts for test

Hi Lian, it appears that you need to have spark-avro_2.11:2.4.4 in your classpath. On Thu, May 21, 2020 at 10:04 AM Lian Jiang wrote: > Thanks Balaji. > > My unit test failed due to dependency incompatibility. Any idea will be > highly appreciated! > > > The test is copied from hudi quick

Re: hudi dependency conflicts for test

Hi, you may also need org.apache.spark:spark-avro_2.11:2.4.4 as a dependency, as you see on the spark-shell example.. Thanks Vinoth On Thu, May 21, 2020 at 10:04 AM Lian Jiang wrote: > Thanks Balaji. > > My unit test failed due to dependency incompatibility. Any idea will be > highly

Re: hudi dependency conflicts for test

Thanks Balaji. My unit test failed due to dependency incompatibility. Any idea will be highly appreciated! The test is copied from hudi quick start: import org.apache.hudi.QuickstartUtils._ import scala.collection.JavaConversions._ import org.apache.spark.sql.SaveMode._ import

Re: hudi dependency conflicts for test

Thanks for using Hudi. Looking at pom definitions between 0.5.1 and 0.5.2, I don't see any difference that could cause this issue. As it works with 0.5.2, I am assuming you are not blocked. Let us know otherwise. Balaji.VOn Wednesday, May 20, 2020, 01:17:08 PM PDT, Lian Jiang wrote:

Re: hudi dependency conflicts for test

Thanks Vinoth. Below dependency has no conflict: compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.3.0' compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.3.0' compile group: 'org.scala-lang', name: 'scala-library', version: '2.11.11' compile group:

Re: hudi dependency conflicts for test

Hi Leon, Sorry for the late reply. Seems like a version mismatch for mockito.. I see you are already trying to exclude it though.. Could you share the full stack trace? On Mon, May 18, 2020 at 1:12 PM Lian Jiang wrote: > Hi, > > I am using hudi in a scala gradle project: > > dependencies {

hudi dependency conflicts for test

Hi, I am using hudi in a scala gradle project: dependencies { compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.4.4' compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.4.4' compile group: 'org.scala-lang', name: 'scala-library', version: