Re: Thoughts on Cloudpickle Update

On Jan 19, 2018 7:28 PM, "Hyukjin Kwon" wrote: > Is it an option to match the latest version of cloudpickle and still set protocol level 2? IMHO, I think this can be an option but I am not fully sure yet if we should/could go ahead for it within Spark 2.X. I need some

Re: Thoughts on Cloudpickle Update

> Is it an option to match the latest version of cloudpickle and still set protocol level 2? IMHO, I think this can be an option but I am not fully sure yet if we should/could go ahead for it within Spark 2.X. I need some investigations including things about Pyrolite. Let's go ahead with

Re: Thoughts on Cloudpickle Update

So if there are different version of Python on the cluster machines I think that's already unsupported so I'm not worried about that. I'd suggest going to the highest released version since there appear to be some useful fixes between 0.4.2 & 0.5.2 Also lets try to keep track in our commit

Re: Thoughts on Cloudpickle Update

Thanks for all the details and background Hyukjin! Regarding the pickle protocol change, if I understand correctly, it is currently at level 2 in Spark which is good for backwards compatibility for all of Python 2. Choosing HIGHEST_PROTOCOL, which is the default for cloudpickle 0.5.0 and above,

Re: [VOTE] Spark 2.3.0 (RC1)

This vote has failed in favor of a new RC. I'll follow up with a new RC2 as soon as the 3 remaining test/UI blockers are resolved. On 17 January 2018 at 16:38, Sameer Agarwal wrote: > Thanks, will do! > > On 16 January 2018 at 22:09, Holden

Re: [build system] currently experiencing git timeouts when building

quick update: it looks like the timeouts have stopped. github finally got back to me about this, but (again) after they stopped happening. i'll be keeping an eye on this for the next few days and will re-escalate if we start having them again. shane On Tue, Jan 16, 2018 at 1:18 PM, shane

Re: Build timed out for `branch-2.3 (hadoop-2.7)`

this doesn't have anything to do w/the git timeouts... those will timeout the build 10 mins after starting (and failing on the initial fetch call). On Wed, Jan 17, 2018 at 9:51 PM, Sameer Agarwal wrote: > FYI, I ended up bumping the build timeouts from 255 to 275 minutes.

Does Spark and Hive use Same SQL parser : ANTLR

Hi Does hive and spark uses same SQL parser provided by ANTLR . Did they generate the same logical plan . Please help on the same. Regards Pralabh Kumar