I agree with Ismael and I’d wait until the new Spark version will be supported by major BigData distributors.
> On 7 Dec 2018, at 14:57, Vishwas Bm <[email protected]> wrote: > > Hi Ismael, > > We have upgraded the spark to 2.4. > In our setup we had run few basic tests and found it to be pretty stable. > > > Thanks & Regards, > Vishwas > > > On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía <[email protected] > <mailto:[email protected]>> wrote: > Hello Vishwas, > > The spark dependency in the spark runner is provided so you can > already pass the dependencies of spark 2.4 and it should work out of > the box. > > JB did a PR to upgrade the version of Spark in the runner, but maybe > it is worth to wait a bit before merging it, at least until some of > the Big Data distributions has spark 2.4.x support available, so far > nobody has upgraded it (well apart of databricks). > > What do others think, should we move ahead or are you aware of any > issue introduced by version 2.4.0? (Notice that the PR just updates > the version so code compatibility should be ok). > > Ismaël > > On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré <[email protected] > <mailto:[email protected]>> wrote: > > > > Hi Vishwas > > > > Yes, I already started the update. > > > > Regards > > JB > > > > On 06/12/2018 07:39, Vishwas Bm wrote: > > > Hi, > > > > > > Currently I see that the spark version dependency used in Beam is > > > //"2.3.2". > > > As spark 2.4 is released now, is there a plan to upgrade Beam spark > > > dependency ? > > > > > > > > > *Thanks & Regards,* > > > *Vishwas > > > * > > > *Mob : 9164886653* > > > > -- > > Jean-Baptiste Onofré > > [email protected] <mailto:[email protected]> > > http://blog.nanthrax.net <http://blog.nanthrax.net/> > > Talend - http://www.talend.com <http://www.talend.com/>
