RE: [EXT] handling skewness issues
Please share the links if they are publicly available. Otherwise please share the name of the talks. Thank you From: Jules Damji Sent: Monday, April 29, 2019 8:04 PM To: Michael Mansour Cc: rajat kumar ; user@spark.apache.org Subject: Re: [EXT] handling skewness issues Yes, indeed! A few talks in the developer and deep dives address the data skews issue and how to address them. I shall let the group know when the talk sessions are available. Cheers Jules Sent from my iPhone Pardon the dumb thumb typos :) On Apr 29, 2019, at 2:13 PM, Michael Mansour mailto:michael_mans...@symantec.com> > wrote: There were recently some fantastic talks about this at the SparkSummit conference in San Francisco. I suggest you check out the SparkSummit YouTube channel after May 9th for a deep dive into this topic. From: rajat kumar mailto:kumar.rajat20...@gmail.com> > Date: Monday, April 29, 2019 at 9:34 AM To: "user@spark.apache.org <mailto:user@spark.apache.org> " mailto:user@spark.apache.org> > Subject: [EXT] handling skewness issues Hi All, How to overcome skewness issues in spark ? I read that we can add some randomness to key column before join and remove that random part after join. is there any better way ? Above method seems to be a workaround. thanks rajat
Re: [EXT] handling skewness issues
Yes, indeed! A few talks in the developer and deep dives address the data skews issue and how to address them. I shall let the group know when the talk sessions are available. Cheers Jules Sent from my iPhone Pardon the dumb thumb typos :) > On Apr 29, 2019, at 2:13 PM, Michael Mansour > wrote: > > There were recently some fantastic talks about this at the SparkSummit > conference in San Francisco. I suggest you check out the SparkSummit YouTube > channel after May 9th for a deep dive into this topic. > > From: rajat kumar > Date: Monday, April 29, 2019 at 9:34 AM > To: "user@spark.apache.org" > Subject: [EXT] handling skewness issues > > Hi All, > > How to overcome skewness issues in spark ? > > I read that we can add some randomness to key column before join and remove > that random part after join. > > is there any better way ? Above method seems to be a workaround. > > thanks > rajat
Re: [EXT] handling skewness issues
There were recently some fantastic talks about this at the SparkSummit conference in San Francisco. I suggest you check out the SparkSummit YouTube channel after May 9th for a deep dive into this topic. From: rajat kumar Date: Monday, April 29, 2019 at 9:34 AM To: "user@spark.apache.org" Subject: [EXT] handling skewness issues Hi All, How to overcome skewness issues in spark ? I read that we can add some randomness to key column before join and remove that random part after join. is there any better way ? Above method seems to be a workaround. thanks rajat
