Re: [Spark Core]: Adding support for size based partition coalescing

[mhawes]

Hi angers.zhu,

Reviving this thread to say that while it's not ideal (as it recomputes the
last stage) I think the `SizeBasedCoaleaser` solution seems like a good
option. If you don't mind re-raising that PR that would be great.
Alternatively I'm happy to make the PR based on your previous PR?

What do you think?

Matt



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [Spark Catalog API] Support for metadata Backup/Restore

[Tianchen Zhang]

Thanks everyone for the input. Yes it makes sense that metadata
backup/restore should be done outside Spark. We will update the customers
with documentations about how that can be done and leave the
implementations to them.

Thanks,
Tianchen

On Tue, May 11, 2021 at 1:14 AM Mich Talebzadeh 
wrote:

> From my experience of dealing with metadata for other applications like
> Hive if needed an external database for Spark metadata would be useful.
>
> However, the maintenance and upgrade of that database should be external
> to Spark (left to the user) and as usual  some form of reliable API or JDBC
> connection will be needed from Spark to this persistent storage. Maybe a
> NoSQL DB will do.
>
> HTH,
>
> Mich
>
>
>view my Linkedin profile
> 
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Tue, 11 May 2021 at 08:58, Wenchen Fan  wrote:
>
>> That's my expectation as well. Spark needs a reliable catalog.
>> backup/restore is just implementation details about how you make your
>> catalog reliable, which should be transparent to Spark.
>>
>> On Sat, May 8, 2021 at 6:54 AM ayan guha  wrote:
>>
>>> Just a consideration:
>>>
>>> Is there a value in backup/restore metadata within spark? I would
>>> strongly argue if the metadata is valuable enough and persistent enough,
>>> why dont just use external metastore? It is fairly straightforward process.
>>> Also regardless you are in cloud or not, database bkp is a routine and
>>> established pattern in most organizations.
>>> You can also enhance HA and DR by having replicas across zones and
>>> regions etc etc
>>>
>>> Thoughts?
>>>
>>>
>>>
>>>
>>> On Sat, 8 May 2021 at 7:02 am, Tianchen Zhang 
>>> wrote:
>>>
 For now we are thinking about adding two methods in Catalog API, not
 SQL commands:
 1. spark.catalog.backup, which backs up the current catalog.
 2. spark.catalog.restore(file), which reads the DFS file and recreates
 the entities described in that file.

 Can you please give an example of exposing client APIs to the end users
 in this approach? The users can only call backup or restore, right?

 Thanks,
 Tianchen

 On Fri, May 7, 2021 at 12:27 PM Wenchen Fan 
 wrote:

> If a catalog implements backup/restore, it can easily expose some
> client APIs to the end-users (e.g. REST API), I don't see a strong reason
> to expose the APIs to Spark. Do you plan to add new SQL commands in Spark
> to backup/restore a catalog?
>
> On Tue, May 4, 2021 at 2:39 AM Tianchen Zhang <
> dustinzhang2...@gmail.com> wrote:
>
>> Hi all,
>>
>> Currently the user-facing Catalog API doesn't support backup/restore
>> metadata. Our customers are asking for such functionalities. Here is a
>> usage example:
>> 1. Read all metadata of one Spark cluster
>> 2. Save them into a Parquet file on DFS
>> 3. Read the Parquet file and restore all metadata in another Spark
>> cluster
>>
>> From the current implementation, Catalog API has the list methods
>> (listDatabases, listFunctions, etc.) but they don't return enough
>> information in order to restore an entity (for example, listDatabases 
>> lose
>> "properties" of the database and we need "describe database extended" to
>> get them). And it only supports createTable (not any other entity
>> creations). The only way we can backup/restore an entity is using Spark 
>> SQL.
>>
>> We want to introduce the backup and restore from an API level. We are
>> thinking of doing this simply by adding backup() and restore() in
>> CatalogImpl, as ExternalCatalog already includes all the methods we need 
>> to
>> retrieve and recreate entities. We are wondering if there is any concern 
>> or
>> drawback of this approach. Please advise.
>>
>> Thank you in advance,
>> Tianchen
>>
> --
>>> Best Regards,
>>> Ayan Guha
>>>
>>

Re: [VOTE] Release Spark 2.4.8 (RC4)

[Liang-Chi Hsieh]

The staging repository for this release can be accessed now too:
https://repository.apache.org/content/repositories/orgapachespark-1383/

Thanks for the guidance.


Liang-Chi Hsieh wrote
> Seems it is closed now after clicking close button in the UI. 





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Spark 2.4.8 (RC4)

[Liang-Chi Hsieh]

Seems it is closed now after clicking close button in the UI. 


Sean Owen-2 wrote
> Is there a separate process that pushes to maven central? That's what we
> have to have in the end.
> 
> On Tue, May 11, 2021, 12:31 PM Liang-Chi Hsieh <

> viirya@

> > wrote:
> 
>> I don't know what will happens if I manually close it now.
>>
>> Not sure if the current status cause a problem? If not, maybe leave as it
>> is?
>>
>>
>> Sean Owen-2 wrote
>> > Hm, yes I see it at
>> >
>> http://pool.sks-keyservers.net/pks/lookup?search=0x653c2301fea493ee&fingerprint=on&op=index
>> > but not on keyserver.ubuntu.com for some reason.
>> > What happens if you try to close it again, perhaps even manually in the
>> UI
>> > there? I don't want to click it unless it messes up the workflow
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> -
>> To unsubscribe e-mail: 

> dev-unsubscribe@.apache

>>
>>





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Spark 2.4.8 (RC4)

[Liang-Chi Hsieh]

Oh, I see. We cannot do release on it as it is still open status.
Okay, let me try to close it manually via UI.


Sean Owen-2 wrote
> Is there a separate process that pushes to maven central? That's what we
> have to have in the end.
> 
> On Tue, May 11, 2021, 12:31 PM Liang-Chi Hsieh <

> viirya@

> > wrote:
> 
>> I don't know what will happens if I manually close it now.
>>
>> Not sure if the current status cause a problem? If not, maybe leave as it
>> is?
>>
>>
>> Sean Owen-2 wrote
>> > Hm, yes I see it at
>> >
>> http://pool.sks-keyservers.net/pks/lookup?search=0x653c2301fea493ee&fingerprint=on&op=index
>> > but not on keyserver.ubuntu.com for some reason.
>> > What happens if you try to close it again, perhaps even manually in the
>> UI
>> > there? I don't want to click it unless it messes up the workflow
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> -
>> To unsubscribe e-mail: 

> dev-unsubscribe@.apache

>>
>>





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Spark 2.4.8 (RC4)

[Sean Owen]

Is there a separate process that pushes to maven central? That's what we
have to have in the end.

On Tue, May 11, 2021, 12:31 PM Liang-Chi Hsieh  wrote:

> I don't know what will happens if I manually close it now.
>
> Not sure if the current status cause a problem? If not, maybe leave as it
> is?
>
>
> Sean Owen-2 wrote
> > Hm, yes I see it at
> >
> http://pool.sks-keyservers.net/pks/lookup?search=0x653c2301fea493ee&fingerprint=on&op=index
> > but not on keyserver.ubuntu.com for some reason.
> > What happens if you try to close it again, perhaps even manually in the
> UI
> > there? I don't want to click it unless it messes up the workflow
>
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] Release Spark 2.4.8 (RC4)

[Liang-Chi Hsieh]

I don't know what will happens if I manually close it now.

Not sure if the current status cause a problem? If not, maybe leave as it
is?


Sean Owen-2 wrote
> Hm, yes I see it at
> http://pool.sks-keyservers.net/pks/lookup?search=0x653c2301fea493ee&fingerprint=on&op=index
> but not on keyserver.ubuntu.com for some reason.
> What happens if you try to close it again, perhaps even manually in the UI
> there? I don't want to click it unless it messes up the workflow





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Release Spark 2.4.8 (RC4)

[Mridul Muralidharan]

+1

Signatures, digests, etc check out fine.
Checked out tag and build/tested.

Regards,
Mridul

On Sun, May 9, 2021 at 4:22 PM Liang-Chi Hsieh  wrote:

> Please vote on releasing the following candidate as Apache Spark version
> 2.4.8.
>
> The vote is open until May 14th at 9AM PST and passes if a majority +1 PMC
> votes are cast, with a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 2.4.8
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> There are currently no issues targeting 2.4.8 (try project = SPARK AND
> "Target Version/s" = "2.4.8" AND status in (Open, Reopened, "In Progress"))
>
> The tag to be voted on is v2.4.8-rc4 (commit
> 163fbd2528a18bf062bddf7b7753631a12a369b5):
> https://github.com/apache/spark/tree/v2.4.8-rc4
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.8-rc4-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1383/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v2.4.8-rc4-docs/
>
> The list of bug fixes going into 2.4.8 can be found at the following URL:
> https://s.apache.org/spark-v2.4.8-rc4
>
> This release is using the release script of the tag v2.4.8-rc4.
>
> FAQ
>
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with an out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 2.4.8?
> ===
>
> The current list of open tickets targeted at 2.4.8 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 2.4.8
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] Release Spark 2.4.8 (RC4)

[Sean Owen]

Hm, yes I see it at
http://pool.sks-keyservers.net/pks/lookup?search=0x653c2301fea493ee&fingerprint=on&op=index
but not on keyserver.ubuntu.com for some reason.
What happens if you try to close it again, perhaps even manually in the UI
there? I don't want to click it unless it messes up the workflow

On Tue, May 11, 2021 at 11:34 AM Liang-Chi Hsieh  wrote:

> I did upload my public key in
> https://dist.apache.org/repos/dist/dev/spark/KEYS.
> I also uploaded it to public keyserver before cutting RC1.
>
> I just also try to search the public key and can find it.
>
>
>
> cloud0fan wrote
> > [image: image.png]
> >
> > I checked the log in https://repository.apache.org/#stagingRepositories,
> > seems the gpg key is not uploaded to the public keyserver. Liang-Chi can
> > you take a look?
>
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [VOTE] Release Spark 2.4.8 (RC4)

[Liang-Chi Hsieh]

I did upload my public key in
https://dist.apache.org/repos/dist/dev/spark/KEYS.
I also uploaded it to public keyserver before cutting RC1.

I just also try to search the public key and can find it.



cloud0fan wrote
> [image: image.png]
> 
> I checked the log in https://repository.apache.org/#stagingRepositories,
> seems the gpg key is not uploaded to the public keyserver. Liang-Chi can
> you take a look?





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [Spark Catalog API] Support for metadata Backup/Restore

[Mich Talebzadeh]

>From my experience of dealing with metadata for other applications like
Hive if needed an external database for Spark metadata would be useful.

However, the maintenance and upgrade of that database should be external to
Spark (left to the user) and as usual  some form of reliable API or JDBC
connection will be needed from Spark to this persistent storage. Maybe a
NoSQL DB will do.

HTH,

Mich


   view my Linkedin profile




*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Tue, 11 May 2021 at 08:58, Wenchen Fan  wrote:

> That's my expectation as well. Spark needs a reliable catalog.
> backup/restore is just implementation details about how you make your
> catalog reliable, which should be transparent to Spark.
>
> On Sat, May 8, 2021 at 6:54 AM ayan guha  wrote:
>
>> Just a consideration:
>>
>> Is there a value in backup/restore metadata within spark? I would
>> strongly argue if the metadata is valuable enough and persistent enough,
>> why dont just use external metastore? It is fairly straightforward process.
>> Also regardless you are in cloud or not, database bkp is a routine and
>> established pattern in most organizations.
>> You can also enhance HA and DR by having replicas across zones and
>> regions etc etc
>>
>> Thoughts?
>>
>>
>>
>>
>> On Sat, 8 May 2021 at 7:02 am, Tianchen Zhang 
>> wrote:
>>
>>> For now we are thinking about adding two methods in Catalog API, not SQL
>>> commands:
>>> 1. spark.catalog.backup, which backs up the current catalog.
>>> 2. spark.catalog.restore(file), which reads the DFS file and recreates
>>> the entities described in that file.
>>>
>>> Can you please give an example of exposing client APIs to the end users
>>> in this approach? The users can only call backup or restore, right?
>>>
>>> Thanks,
>>> Tianchen
>>>
>>> On Fri, May 7, 2021 at 12:27 PM Wenchen Fan  wrote:
>>>
 If a catalog implements backup/restore, it can easily expose some
 client APIs to the end-users (e.g. REST API), I don't see a strong reason
 to expose the APIs to Spark. Do you plan to add new SQL commands in Spark
 to backup/restore a catalog?

 On Tue, May 4, 2021 at 2:39 AM Tianchen Zhang <
 dustinzhang2...@gmail.com> wrote:

> Hi all,
>
> Currently the user-facing Catalog API doesn't support backup/restore
> metadata. Our customers are asking for such functionalities. Here is a
> usage example:
> 1. Read all metadata of one Spark cluster
> 2. Save them into a Parquet file on DFS
> 3. Read the Parquet file and restore all metadata in another Spark
> cluster
>
> From the current implementation, Catalog API has the list methods
> (listDatabases, listFunctions, etc.) but they don't return enough
> information in order to restore an entity (for example, listDatabases lose
> "properties" of the database and we need "describe database extended" to
> get them). And it only supports createTable (not any other entity
> creations). The only way we can backup/restore an entity is using Spark 
> SQL.
>
> We want to introduce the backup and restore from an API level. We are
> thinking of doing this simply by adding backup() and restore() in
> CatalogImpl, as ExternalCatalog already includes all the methods we need 
> to
> retrieve and recreate entities. We are wondering if there is any concern 
> or
> drawback of this approach. Please advise.
>
> Thank you in advance,
> Tianchen
>
 --
>> Best Regards,
>> Ayan Guha
>>
>

Re: [Spark Catalog API] Support for metadata Backup/Restore

[Wenchen Fan]

That's my expectation as well. Spark needs a reliable catalog.
backup/restore is just implementation details about how you make your
catalog reliable, which should be transparent to Spark.

On Sat, May 8, 2021 at 6:54 AM ayan guha  wrote:

> Just a consideration:
>
> Is there a value in backup/restore metadata within spark? I would strongly
> argue if the metadata is valuable enough and persistent enough, why dont
> just use external metastore? It is fairly straightforward process. Also
> regardless you are in cloud or not, database bkp is a routine and
> established pattern in most organizations.
> You can also enhance HA and DR by having replicas across zones and regions
> etc etc
>
> Thoughts?
>
>
>
>
> On Sat, 8 May 2021 at 7:02 am, Tianchen Zhang 
> wrote:
>
>> For now we are thinking about adding two methods in Catalog API, not SQL
>> commands:
>> 1. spark.catalog.backup, which backs up the current catalog.
>> 2. spark.catalog.restore(file), which reads the DFS file and recreates
>> the entities described in that file.
>>
>> Can you please give an example of exposing client APIs to the end users
>> in this approach? The users can only call backup or restore, right?
>>
>> Thanks,
>> Tianchen
>>
>> On Fri, May 7, 2021 at 12:27 PM Wenchen Fan  wrote:
>>
>>> If a catalog implements backup/restore, it can easily expose some client
>>> APIs to the end-users (e.g. REST API), I don't see a strong reason to
>>> expose the APIs to Spark. Do you plan to add new SQL commands in Spark to
>>> backup/restore a catalog?
>>>
>>> On Tue, May 4, 2021 at 2:39 AM Tianchen Zhang 
>>> wrote:
>>>
 Hi all,

 Currently the user-facing Catalog API doesn't support backup/restore
 metadata. Our customers are asking for such functionalities. Here is a
 usage example:
 1. Read all metadata of one Spark cluster
 2. Save them into a Parquet file on DFS
 3. Read the Parquet file and restore all metadata in another Spark
 cluster

 From the current implementation, Catalog API has the list methods
 (listDatabases, listFunctions, etc.) but they don't return enough
 information in order to restore an entity (for example, listDatabases lose
 "properties" of the database and we need "describe database extended" to
 get them). And it only supports createTable (not any other entity
 creations). The only way we can backup/restore an entity is using Spark 
 SQL.

 We want to introduce the backup and restore from an API level. We are
 thinking of doing this simply by adding backup() and restore() in
 CatalogImpl, as ExternalCatalog already includes all the methods we need to
 retrieve and recreate entities. We are wondering if there is any concern or
 drawback of this approach. Please advise.

 Thank you in advance,
 Tianchen

>>> --
> Best Regards,
> Ayan Guha
>

Re: [VOTE] Release Spark 2.4.8 (RC4)

[Wenchen Fan]

[image: image.png]

I checked the log in https://repository.apache.org/#stagingRepositories,
seems the gpg key is not uploaded to the public keyserver. Liang-Chi can
you take a look?

On Tue, May 11, 2021 at 3:47 PM Wenchen Fan  wrote:

> +1
>
> On Tue, May 11, 2021 at 2:59 AM Holden Karau  wrote:
>
>> +1 - pip install with Py 2.7 works (with the understandable warnings
>> regarding Python 2.7 no longer being maintained).
>>
>> On Mon, May 10, 2021 at 11:18 AM sarutak  wrote:
>> >
>> > +1 (non-binding)
>> >
>> > - Kousuke
>> >
>> > > It looks like the repository is "open" - it doesn't publish until
>> > > "closed" after all artifacts are uploaded. Is that it?
>> > > Otherwise +1 from me.
>> > >
>> > > On Mon, May 10, 2021 at 1:10 AM Liang-Chi Hsieh 
>> > > wrote:
>> > >
>> > >> Yea, I don't know why it happens.
>> > >>
>> > >> I remember RC1 also has the same issue. But RC2 and RC3 don't.
>> > >>
>> > >> Does it affect the RC?
>> > >>
>> > >> John Zhuge wrote
>> > >>> Got this error when browsing the staging repository:
>> > >>>
>> > >>> 404 - Repository "orgapachespark-1383 (staging: open)"
>> > >>> [id=orgapachespark-1383] exists but is not exposed.
>> > >>>
>> > >>> John Zhuge
>> > >>
>> > >> --
>> > >> Sent from:
>> > >> http://apache-spark-developers-list.1001551.n3.nabble.com/
>> > >>
>> > >>
>> > > -
>> > >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >
>> > -
>> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >
>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>

Re: [VOTE] Release Spark 2.4.8 (RC4)

[Wenchen Fan]

+1

On Tue, May 11, 2021 at 2:59 AM Holden Karau  wrote:

> +1 - pip install with Py 2.7 works (with the understandable warnings
> regarding Python 2.7 no longer being maintained).
>
> On Mon, May 10, 2021 at 11:18 AM sarutak  wrote:
> >
> > +1 (non-binding)
> >
> > - Kousuke
> >
> > > It looks like the repository is "open" - it doesn't publish until
> > > "closed" after all artifacts are uploaded. Is that it?
> > > Otherwise +1 from me.
> > >
> > > On Mon, May 10, 2021 at 1:10 AM Liang-Chi Hsieh 
> > > wrote:
> > >
> > >> Yea, I don't know why it happens.
> > >>
> > >> I remember RC1 also has the same issue. But RC2 and RC3 don't.
> > >>
> > >> Does it affect the RC?
> > >>
> > >> John Zhuge wrote
> > >>> Got this error when browsing the staging repository:
> > >>>
> > >>> 404 - Repository "orgapachespark-1383 (staging: open)"
> > >>> [id=orgapachespark-1383] exists but is not exposed.
> > >>>
> > >>> John Zhuge
> > >>
> > >> --
> > >> Sent from:
> > >> http://apache-spark-developers-list.1001551.n3.nabble.com/
> > >>
> > >>
> > > -
> > >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >
> > -
> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >
>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: Bintray replacement for spark-packages.org

[Dongjoon Hyun]

Thank you, Yi and all.

Then, after 2.4.8 release, shall we start to roll 3.1.2 and 3.0.3.

Bests,
Dongjoon.

On Mon, May 10, 2021 at 10:50 PM Yi Wu  wrote:

> Hi wenchen,
>
> I'd like to volunteer for Apache Spark 3.0.3 release.
>
> Thanks,
> Yi
>
> On Fri, Apr 30, 2021 at 12:37 AM Dongjoon Hyun 
> wrote:
>
>> I agree with Wenchen.
>>
>> I can volunteer for Apache Spark 3.1.2 release manager at least.
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Wed, Apr 28, 2021 at 10:15 AM Wenchen Fan  wrote:
>>
>>> Shall we make new releases for 3.0 and 3.1? So that people don't need to
>>> change the sbt resolver/pom files to work around Bintray sunset. It's also
>>> been a while since the last 3.0 and 3.1 releases.
>>>
>>> On Tue, Apr 27, 2021 at 9:02 AM Matthew Powers <
>>> matthewkevinpow...@gmail.com> wrote:
>>>
 Great job fixing this!!  I just checked and it's working on my end.  
 Updated
 the resolver
 
 and sbt test still works just fine.

 On Mon, Apr 26, 2021 at 3:31 AM Bo Zhang 
 wrote:

> Hi Apache Spark devs,
>
> As you might know, Bintray, which is the repository service used for
> spark-packages.org, is in its sunset process. There was a planned
> brown-out on April 12th
>  and there will be
> another one on April 26th
> , and it will no
> longer be available from May 1st.
>
> We have spun up a new repository service at
> https://repos.spark-packages.org and it will be the new home for the
> artifacts on spark-packages.
>
> Given the planned Bintray brown-out, this is a good time for us to
> test the new repository service. To consume artifacts from that, please
> replace "dl.bintray.com/spark-packages/maven" with "
> repos.spark-packages.org" in the Maven pom files or sbt build files
> in your repositories, e.g.: https://github.com/apache/spark/pull/32346
>
> We are still working on the release process to the new repository
> service, and will provide an update here shortly.
>
> If you have any questions for using the new repository service, or any
> general questions for spark-packages, please reach out to
> feedb...@spark-packages.org.
>
> Thanks,
> Bo
>