It seems that Cloudera has it now, not sure if worth to wait for the Hortonworks maybe worth waiting for EMR. https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.11/
A pro move to Spark 2.4.0 argument is for the future oriented (non hadoop friends), because the support for kubernetes has improved a lot in this release. On Fri, Dec 7, 2018 at 4:56 PM David Morávek <[email protected]> wrote: > > +1 for waiting for HDP and CDH adoption > > Sent from my iPhone > > On 7 Dec 2018, at 16:38, Alexey Romanenko <[email protected]> wrote: > > I agree with Ismael and I’d wait until the new Spark version will be > supported by major BigData distributors. > > On 7 Dec 2018, at 14:57, Vishwas Bm <[email protected]> wrote: > > Hi Ismael, > > We have upgraded the spark to 2.4. > In our setup we had run few basic tests and found it to be pretty stable. > > > Thanks & Regards, > Vishwas > > > On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía <[email protected]> wrote: >> >> Hello Vishwas, >> >> The spark dependency in the spark runner is provided so you can >> already pass the dependencies of spark 2.4 and it should work out of >> the box. >> >> JB did a PR to upgrade the version of Spark in the runner, but maybe >> it is worth to wait a bit before merging it, at least until some of >> the Big Data distributions has spark 2.4.x support available, so far >> nobody has upgraded it (well apart of databricks). >> >> What do others think, should we move ahead or are you aware of any >> issue introduced by version 2.4.0? (Notice that the PR just updates >> the version so code compatibility should be ok). >> >> Ismaël >> >> On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré <[email protected]> >> wrote: >> > >> > Hi Vishwas >> > >> > Yes, I already started the update. >> > >> > Regards >> > JB >> > >> > On 06/12/2018 07:39, Vishwas Bm wrote: >> > > Hi, >> > > >> > > Currently I see that the spark version dependency used in Beam is >> > > //"2.3.2". >> > > As spark 2.4 is released now, is there a plan to upgrade Beam spark >> > > dependency ? >> > > >> > > >> > > *Thanks & Regards,* >> > > *Vishwas >> > > * >> > > *Mob : 9164886653* >> > >> > -- >> > Jean-Baptiste Onofré >> > [email protected] >> > http://blog.nanthrax.net >> > Talend - http://www.talend.com > >
