preferredlocations for hadoopfsrelations based baseRelations

HI Spark developers, I have created new format extending fileformat. I see getPrefferedLocations is available if newCustomRDD is created. Since fileformat is based off FileScanRDD which uses readfile method to read partitioned file, Is there a way to add desired preferredLocations ? Appreciate

Re: [VOTE] Release Spark 2.4.6 (RC8)

If this is something we expect to mostly impact new users I think we can push them towards Spark 3 instead of introducing a behaviour change in 2.4.6 On Wed, Jun 3, 2020 at 12:34 PM Mridul Muralidharan wrote: > > Is this a behavior change in 2.4.x from earlier version ? > Or are we proposing t

Re: [VOTE] Release Spark 2.4.6 (RC8)

Is this a behavior change in 2.4.x from earlier version ? Or are we proposing to introduce a functionality to help with adoption ? Regards, Mridul On Wed, Jun 3, 2020 at 10:32 AM Xiao Li wrote: > Yes. Spark 3.0 RC2 works well. > > I think the current behavior in Spark 2.4 affects the adopti

Re: [VOTE] Release Spark 2.4.6 (RC8)

Yes. Spark 3.0 RC2 works well. I think the current behavior in Spark 2.4 affects the adoption, especially for the new users who want to try Spark in their local environment. It impacts all our built-in clients, like Scala Shell and PySpark. Should we consider back-porting it to 2.4? Although thi

Re: [VOTE] Release Spark 2.4.6 (RC8)

I believe that was fixed in 3.0 and there was a decision not to backport the fix: SPARK-31170 On Wed, Jun 3, 2020 at 1:04 PM Xiao Li wrote: > Just downloaded it in my local macbook. Trying to create a table using the > pre-built PySpark. It sou

Re: [VOTE] Release Spark 2.4.6 (RC8)

Just downloaded it in my local macbook. Trying to create a table using the pre-built PySpark. It sounds like the conf "spark.sql.warehouse.dir" does not take an effect. It is trying to create a directory in "file:/user/hive/warehouse/t1". I have not done any investigation yet. Have any of you hit t

Re: [VOTE] Release Spark 2.4.6 (RC8)

+1 Bests, Dongjoon On Wed, Jun 3, 2020 at 5:59 AM Tom Graves wrote: > +1 > > Tom > > On Sunday, May 31, 2020, 06:47:09 PM CDT, Holden Karau < > hol...@pigscanfly.ca> wrote: > > > Please vote on releasing the following candidate as Apache Spark > version 2.4.6. > > The vote is open until June 5

Re: [VOTE] Release Spark 2.4.6 (RC8)

 +1 Tom On Sunday, May 31, 2020, 06:47:09 PM CDT, Holden Karau wrote: Please vote on releasing the following candidate as Apache Spark version 2.4.6. The vote is open until June 5th at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Releas