Re: [VOTE] SPARK 2.4.0 (RC3)

Hi, Sorry if it's stupid question, but where can I find the release notes of 2.4.0? jg > On Oct 10, 2018, at 2:00 PM, Imran Rashid > wrote: > > Sorry I had messed up my testing earlier, so I only just discovered >

Re: [VOTE] SPARK 2.4.0 (RC3)

=12315420=12342385 > > <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420=12342385> > > Bests, > Dongjoon. > > > On Wed, Oct 10, 2018 at 11:29 AM Jean Georges Perrin <mailto:j...@jgp.net>> wrote: > Hi, > > Sorry if it's s

Re: [RESULT] [VOTE] Functional DataSourceV2 in Spark 3.0

Hi, I am kind of new at the whole Apache process (not specifically Spark). Does that means that the DataSourceV2 is dead or stays experimental? Thanks for clarifying for a newbie. jg > On Mar 3, 2019, at 11:21, Ryan Blue wrote: > > This vote fails with the following counts: > > 3 +1

Static functions

Hey guys, We have 381 static functions now (including the deprecated). I am trying to sort them out by group/tag them. So far, I have: Array Conversion Date Math Trigo (sub group of maths) Security Streaming String Technical Do you see more categories? Tags? Thanks! jg — Jean Georges Perrin

Re: Static functions

on the Javadoc in https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/functions.html… or anywhere else (Scala doc or SQL functions). Jean Georges Perrin j...@jgp.net > On Feb 11, 2019, at 09:42, Jacek Laskowski wrote: > > Hi Jean, > > I thought the functions have alre

Re: DataSourceV2 sync, 17 April 2019

” v2 API that will be available after the release of Spark v3. jg -- Jean Georges Perrin j...@jgp.net > On Apr 19, 2019, at 10:10, Ryan Blue wrote: > > Here are my notes from the last DSv2 sync. As always: > > If you’d like to attend the sync, send me an email and I’ll add you t

Spark in Action, 2e...

://linkedin.com/in/jgperrin <http://linkedin.com/in/jgperrin>) Thanks for building Spark! Jean-Georges Perrin j...@jgp.net

Re: [DISCUSSION]JDK11 for Apache 2.x?

Not a contributor, but a user perspective… As Spark 3.x will be an evolution, I am not completely shocked that it would imply a Java 11 requirement as well. Would be great to have both Java 8 and Java 11, but one needs to be able to say goodbye. Java 8 is great, still using it actively in

Re: Thoughts on Spark 3 release, or a preview release

As a user/non committer, +1 I love the idea of an early 3.0.0 so we can test current dev against it, I know the final 3.x will probably need another round of testing when it gets out, but less for sure... I know I could checkout and compile, but having a “packaged” preversion is great if it

Issues with Delta Lake on 3.0.0 preview + preview 2

Hi there, Trying to run a very simple app saving content of a dataframe to Delta Lake. Code works great on 2.4.4 but fails on 3.0.0 preview & preview 2. Tried on both Delta Lake 0.5.0 and 0.4.0. Code (I know, it’s amazing): df.write().format("delta") .mode("overwrite")

Issue with map Java lambda function with 3.0.0 preview and preview 2

Hey guys, This code: Dataset incrementalDf = spark .createDataset(l, Encoders.INT()) .toDF(); Dataset dotsDs = incrementalDf .map(status -> { double x = Math.random() * 2 - 1; double y = Math.random() * 2 - 1; counter++; if

Re: Issue with map Java lambda function with 3.0.0 preview and preview 2

I forgot… it does the same thing with the reducer… int dartsInCircle = dotsDs.reduce((x, y) -> x + y); jg > On Dec 28, 2019, at 12:38 PM, Jean-Georges Perrin wrote: > > Hey guys, > > This code: > > Dataset incrementalDf = spark > .cr

Re: Issue with map Java lambda function with 3.0.0 preview and preview 2

rent in Spark 3; some of that has always > been the case with Java 8 in Spark 2. I think it might be related to > Scala 2.12; were you using Spark 2 with Scala 2.11 before? > > On Sat, Dec 28, 2019 at 11:38 AM Jean-Georges Perrin wrote: >> >> Hey guys, >> &

Re: Data Contracts

Hi, While I was at PayPal, we open sourced a template of Data Contract, it is here: https://github.com/paypal/data-contract-template. Companies like GX (Great Expectations) are interested in using it. Spark could read some elements form it pretty easily, like schema validation, some rules