To clarify Ismaël's comment Cloudera repo indicates Cloudera 6.1 will have spark 2.4 but CDH is currently still on 6.0.
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.11/2.4.0-cdh6.1.0/ With the HWX / Cloudera merger the release cycle is not announced but 6.1 will likely not be there until early 2019. Also note that many folk will still be on CDH 5 as CDH 6 is relatively new. On Fri, Dec 7, 2018 at 5:06 PM Ismaël Mejía <[email protected]> wrote: > It seems that Cloudera has it now, not sure if worth to wait for the > Hortonworks maybe worth waiting for EMR. > > https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.11/ > > A pro move to Spark 2.4.0 argument is for the future oriented (non > hadoop friends), because the support for kubernetes has improved a lot > in this release. > > On Fri, Dec 7, 2018 at 4:56 PM David Morávek <[email protected]> > wrote: > > > > +1 for waiting for HDP and CDH adoption > > > > Sent from my iPhone > > > > On 7 Dec 2018, at 16:38, Alexey Romanenko <[email protected]> > wrote: > > > > I agree with Ismael and I’d wait until the new Spark version will be > supported by major BigData distributors. > > > > On 7 Dec 2018, at 14:57, Vishwas Bm <[email protected]> wrote: > > > > Hi Ismael, > > > > We have upgraded the spark to 2.4. > > In our setup we had run few basic tests and found it to be pretty stable. > > > > > > Thanks & Regards, > > Vishwas > > > > > > On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía <[email protected]> wrote: > >> > >> Hello Vishwas, > >> > >> The spark dependency in the spark runner is provided so you can > >> already pass the dependencies of spark 2.4 and it should work out of > >> the box. > >> > >> JB did a PR to upgrade the version of Spark in the runner, but maybe > >> it is worth to wait a bit before merging it, at least until some of > >> the Big Data distributions has spark 2.4.x support available, so far > >> nobody has upgraded it (well apart of databricks). > >> > >> What do others think, should we move ahead or are you aware of any > >> issue introduced by version 2.4.0? (Notice that the PR just updates > >> the version so code compatibility should be ok). > >> > >> Ismaël > >> > >> On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré <[email protected]> > wrote: > >> > > >> > Hi Vishwas > >> > > >> > Yes, I already started the update. > >> > > >> > Regards > >> > JB > >> > > >> > On 06/12/2018 07:39, Vishwas Bm wrote: > >> > > Hi, > >> > > > >> > > Currently I see that the spark version dependency used in Beam is > >> > > //"2.3.2". > >> > > As spark 2.4 is released now, is there a plan to upgrade Beam spark > >> > > dependency ? > >> > > > >> > > > >> > > *Thanks & Regards,* > >> > > *Vishwas > >> > > * > >> > > *Mob : 9164886653* > >> > > >> > -- > >> > Jean-Baptiste Onofré > >> > [email protected] > >> > http://blog.nanthrax.net > >> > Talend - http://www.talend.com > > > > >
