+1 for waiting for HDP and CDH adoption Sent from my iPhone
> On 7 Dec 2018, at 16:38, Alexey Romanenko <[email protected]> wrote: > > I agree with Ismael and I’d wait until the new Spark version will be > supported by major BigData distributors. > >> On 7 Dec 2018, at 14:57, Vishwas Bm <[email protected]> wrote: >> >> Hi Ismael, >> >> We have upgraded the spark to 2.4. >> In our setup we had run few basic tests and found it to be pretty stable. >> >> >> Thanks & Regards, >> Vishwas >> >> >>> On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía <[email protected]> wrote: >>> Hello Vishwas, >>> >>> The spark dependency in the spark runner is provided so you can >>> already pass the dependencies of spark 2.4 and it should work out of >>> the box. >>> >>> JB did a PR to upgrade the version of Spark in the runner, but maybe >>> it is worth to wait a bit before merging it, at least until some of >>> the Big Data distributions has spark 2.4.x support available, so far >>> nobody has upgraded it (well apart of databricks). >>> >>> What do others think, should we move ahead or are you aware of any >>> issue introduced by version 2.4.0? (Notice that the PR just updates >>> the version so code compatibility should be ok). >>> >>> Ismaël >>> >>> On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré <[email protected]> >>> wrote: >>> > >>> > Hi Vishwas >>> > >>> > Yes, I already started the update. >>> > >>> > Regards >>> > JB >>> > >>> > On 06/12/2018 07:39, Vishwas Bm wrote: >>> > > Hi, >>> > > >>> > > Currently I see that the spark version dependency used in Beam is >>> > > //"2.3.2". >>> > > As spark 2.4 is released now, is there a plan to upgrade Beam spark >>> > > dependency ? >>> > > >>> > > >>> > > *Thanks & Regards,* >>> > > *Vishwas >>> > > * >>> > > *Mob : 9164886653* >>> > >>> > -- >>> > Jean-Baptiste Onofré >>> > [email protected] >>> > http://blog.nanthrax.net >>> > Talend - http://www.talend.com >
