Re: Run Python User Defined Functions / code in Spark with Scala Codebase

Hello Jayant, Thanks for great OSS Contribution :) On Thu, Jul 12, 2018 at 1:36 PM, Jayant Shekhar wrote: > Hello Chetan, > > Sorry missed replying earlier. You can find some sample code here : > > http://sparkflows.readthedocs.io/en/latest/user-guide/ > python/pipe-python.html > > We will

Stale PR update and review request

Hi all, I was checking https://spark-prs.appspot.com/users who has PRs more then 10. viirya 13 mgaido91 12 wangyum 12 maropu

Re: [VOTE] SPARK 2.3.2 (RC3)

Hi Sean, I just did a clean build with mvn/sbt on 2.3.2, I didn't meet the errors you pasted here. I'm not sure how it happens. Sean Owen 于2018年7月16日周一 上午6:30写道： > Looks good to me, with the following caveats. > > First see the discussion on > https://issues.apache.org/jira/browse/SPARK-24813

Re: [VOTE] SPARK 2.3.2 (RC3)

Looks good to me, with the following caveats. First see the discussion on https://issues.apache.org/jira/browse/SPARK-24813 ; the flaky HiveExternalCatalogVersionsSuite will probably fail all the time right now. That's not a regression and is a test-only issue, so don't think it must block the

Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

Makes sense. Thanks for looking into this. On Sun, Jul 15, 2018 at 1:51 PM Sean Owen wrote: > Yesterday I cleaned out old Spark releases from the mirror system -- we're > supposed to only keep the latest release from active branches out on > mirrors. (All releases are available from the Apache

Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

Yesterday I cleaned out old Spark releases from the mirror system -- we're supposed to only keep the latest release from active branches out on mirrors. (All releases are available from the Apache archive site.) Having done so I realized quickly that the HiveExternalCatalogVersionsSuite relies on

Re: [VOTE] SPARK 2.3.2 (RC3)

+1. The Spark 2.3 regressions I'm aware of are all fixed. On Sun, Jul 15, 2018 at 4:09 PM Saisai Shao wrote: > Please vote on releasing the following candidate as Apache Spark version > 2.3.2. > > The vote is open until July 20 PST and passes if a majority +1 PMC votes > are cast, with a

Re: Pyspark access to scala/java libraries

If you want to see some examples in a library shows a way to do it - https://github.com/sparklingpandas/sparklingml and high performance spark also talks about it. On Sun, Jul 15, 2018, 11:57 AM <0xf0f...@protonmail.com.invalid> wrote: > Check >

Re: Pyspark access to scala/java libraries

Check https://stackoverflow.com/questions/31684842/calling-java-scala-function-from-a-task ​Sent with ProtonMail Secure Email.​ ‐‐‐ Original Message ‐‐‐ On July 15, 2018 8:01 AM, Mohit Jaggi wrote: > Trying again…anyone know how to make this work? > > > On Jul 9, 2018, at 3:45 PM,

[VOTE] SPARK 2.3.2 (RC3)

Please vote on releasing the following candidate as Apache Spark version 2.3.2. The vote is open until July 20 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.3.2 [ ] -1 Do not release this package because ... To

Re: Pyspark access to scala/java libraries

Trying again…anyone know how to make this work? > On Jul 9, 2018, at 3:45 PM, Mohit Jaggi wrote: > > Folks, > I am writing some Scala/Java code and want it to be usable from pyspark. > > For example: > class MyStuff(addend: Int) { > def myMapFunction(x: Int) = x + addend > } > > I want