Re: [Action required] Default Spark profile changed to 3.2

[Y Ethan Guo]

There is a hotfix (https://github.com/apache/hudi/pull/8822) merged
recently to fix the default build.  SPARK_HOME does not matter and the
latest master builds on my end with Spark 3.3 as SPARK_HOME.

On Fri, Jun 2, 2023 at 1:28 AM Vinoth Chandar  wrote:

> Hi,
>
> Just tried doing a mvn clean install -DskipTests, and the build failed. My
> local SPARK_HOME is pointing to spark 3.3 installation.
> Does that all matter now? Quite possible this is an issue with my setup,
> just flagging.
>
> Thanks
> Vinoth
>
> On Fri, May 26, 2023 at 8:30 AM Shiyan Xu 
> wrote:
>
> > Hi all,
> >
> > We recently landed a change
> > <
> >
> https://github.com/apache/hudi/commit/516c3d59404934e6a142ea1c9d97002c065f8a4f
> > >
> > in master switching the default Spark profile from 2.4 to 3.2. If your
> > local Hudi repo is configured to use Spark 2.4, you may need to re-import
> > the IDEA project (this may involve clearing `.idea/` folder and *.iml
> files
> > in each maven module)
> >
> > I'd also like to acknowledge contributions from Forward Xu, Rahil, Zhang
> > Yue, and Danny, who have previously worked on or helped tackle this
> > migration, as it involved a lot of dependency issue wrangling and test
> > fixes.
> >
> > Cheers,
> >
> > --
> > Best,
> > Shiyan
> >
>

Re: [Action required] Default Spark profile changed to 3.2

[Vinoth Chandar]

Hi,

Just tried doing a mvn clean install -DskipTests, and the build failed. My
local SPARK_HOME is pointing to spark 3.3 installation.
Does that all matter now? Quite possible this is an issue with my setup,
just flagging.

Thanks
Vinoth

On Fri, May 26, 2023 at 8:30 AM Shiyan Xu 
wrote:

> Hi all,
>
> We recently landed a change
> <
> https://github.com/apache/hudi/commit/516c3d59404934e6a142ea1c9d97002c065f8a4f
> >
> in master switching the default Spark profile from 2.4 to 3.2. If your
> local Hudi repo is configured to use Spark 2.4, you may need to re-import
> the IDEA project (this may involve clearing `.idea/` folder and *.iml files
> in each maven module)
>
> I'd also like to acknowledge contributions from Forward Xu, Rahil, Zhang
> Yue, and Danny, who have previously worked on or helped tackle this
> migration, as it involved a lot of dependency issue wrangling and test
> fixes.
>
> Cheers,
>
> --
> Best,
> Shiyan
>

[Action required] Default Spark profile changed to 3.2

[Shiyan Xu]

Hi all,

We recently landed a change

in master switching the default Spark profile from 2.4 to 3.2. If your
local Hudi repo is configured to use Spark 2.4, you may need to re-import
the IDEA project (this may involve clearing `.idea/` folder and *.iml files
in each maven module)

I'd also like to acknowledge contributions from Forward Xu, Rahil, Zhang
Yue, and Danny, who have previously worked on or helped tackle this
migration, as it involved a lot of dependency issue wrangling and test
fixes.

Cheers,

-- 
Best,
Shiyan