Re: [VOTE][SPARK-25299] SPIP: Shuffle Storage API

e SPIP document here [docs.google.com] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1d6egnL6WHOwWZe8MWv3m8n4PToNacdx7n-5F0iMSWwhCQ_edit=DwMFJg=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc=UG2t14gfU8QHfoj4tUD__9bIVg1xxTM3R8GHmvMUXTU=rCSgQGD6L4of4oa0QxiTJ8IPaVdGlZVarhA4-QvO80Q=> >> . >> >> >> >> The discussion thread for the SPIP was conducted here [lists.apache.org] >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_2fe82b6b86daadb1d2edaef66a2d1c4dd2f45449656098ee38c50079-40-253Cdev.spark.apache.org-253E=DwMFJg=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc=UG2t14gfU8QHfoj4tUD__9bIVg1xxTM3R8GHmvMUXTU=kSJizQH7v4OHG6D7aVsLA-m0ApZxOa24CzHZv1EzLxg=> >> . >> >> >> >> Please vote on whether or not this proposal is agreeable to you. >> >> >> >> Thanks! >> >> >> >> -Matt Cheah >> >> >> >> >> -- >> >> Ryan Blue >> >> Software Engineer >> >> Netflix >> >> > > -- > John > -- -Dhruve Ashar

Re: [VOTE] Release Apache Spark 2.4.1 (RC8)

ith the fix, > but understand the tradeoffs here. > > On Wed, Mar 20, 2019 at 10:17 AM Sean Owen wrote: > >> Is it a regression from 2.4.0? that's not the only criteria but part of >> it. >> The version link is >> https://issues.apache.org/jira/projects/SPARK/

Re: [VOTE] Release Apache Spark 2.4.1 (RC8)

to make timely releases, we will typically not hold the >> > release unless the bug in question is a regression from the previous >> > release. That being said, if there is something which is a regression >> > that has not been correctly targeted please ping me or a committer to >> > help target the issue. >> > >> > >> > DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, >> Inc >> > >> > >> > - >> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> > >> > >> > >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> > > -- > Stavros Kontopoulos > > *Senior Software Engineer* > *Lightbend, Inc.* > > *p: +30 6977967274 <%2B1%20650%20678%200020>* > *e: stavros.kontopou...@lightbend.com* > > > -- -Dhruve Ashar

Checkpointing in Spark - Cleaning files and Support across app attempts

While checkpointing RDDs as a part of an application that doesn't use spark-streaming, I observed that the checkpointed files are not being cleaned up even after the application completes successfully. Is it because we assume that checkpointing would be primarily used for spark-streaming

Re: Setting YARN executors' JAVA_HOME

on't appear to set JAVA_HOME (and maybe > aren't even supposed to?). > > The 1.0.1 docs > <https://spark.apache.org/docs/1.0.1/running-on-yarn.html#environment-variables> > are the last ones to reference the SPARK_YARN_USER_ENV var, afaict. > > What's the preferred way of passing YARN a custom JAVA_HOME that will be > applied to executors' containers? > > Thanks! > -- -Dhruve Ashar

Re: Welcoming Felix Cheung as a committer

to SparkR and we're excited to have him join >> officially. Congrats and welcome, Felix! >> > >> > Matei >> > - >> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> > >> >> - >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> -- -Dhruve Ashar

Re: Question about equality of o.a.s.sql.Row

rk-2.0.0-preview-docs/api/scala/index.html#org.apache.spark.sql.Row > > > Pull requests for documentation welcome! > > >> (3) If (3) is expected, is there any recommendation to write code of >> equality of two Rows that have an Array or complex types (e.g. Map)? > > > Internally for tests, we usually compare the string representation of the > Row. > -- -Dhruve Ashar

Re: SBT doesn't pick resource file after clean

ue / "foo.properties") > > On Thu, May 19, 2016 at 7:21 PM, dhruve ashar <dhruveas...@gmail.com> > wrote: > > Based on the conversation on PR, the intent was not to pollute the source > > directory and hence we are placing the generated file outside it in th

Re: SBT doesn't pick resource file after clean

ou need to make the "compile" task of the appropriate module > > depend on the task that generates the resource file? > > > > Sorry but my knowledge of sbt doesn't really go too far. > > > > On Tue, May 17, 2016 at 11:58 AM, dhruve ashar <dhruveas...@gmail.com&

SBT doesn't pick resource file after clean

We are trying to pick the spark version automatically from pom instead of manually modifying the files. This also includes richer pieces of information like last commit, version, user who built the code etc to better identify the framework running. The setup is as follows : - A shell script