Re: [apache/spark] [SPARK-29674][CORE] Update dropwizard metrics to 4.1.x for JDK 9+ (#26332)

[Dongjoon Hyun]

The Ganglia module has only 2 files.
In addition to dropping, we may choose the following two ways to support it
still partially
like `kafka-0.8` which Apache Spark supports in Scala 2.11 only.

   1. We can stick to `dropwizard 3.x` for JDK8 (by default) and use
`dropwizard 4.x` for `hadoop-3.2` profile only.
   2. If we upgrade to `drop wizard 4.x` completely, we can make the
Ganglia module as an external packages (with dropwizard 3.x) for Apache
Spark 3.0 JDK8.

$ tree .
.
├── pom.xml
└── src
└── main
└── scala
└── org
└── apache
└── spark
└── metrics
└── sink
└── GangliaSink.scala

---
Language files  blankcomment
code
---
Scala1 20 17
  59
Maven1  4 17
  27
---
SUM: 2 24 34
  86
---

Bests,
Dongjoon.


On Wed, Oct 30, 2019 at 6:18 PM Sean Owen  wrote:

> I wanted to raise this to dev@.
>
> So, updating dropwizard metrics from 3.2.x to 4.x might be important for
> JDK 11 support. Our tests pass as-is without this update. But we don't test
> some elements of this metrics support, like Ganglia integration. And I have
> heard reports that downstream custom usages of dropwizard 3.2.x doesn't
> work on JDK 11.
>
> The bad news is that the Ganglia integration doesn't exist anymore in 4.x.
> And we have a whole custom module for that integration with Spark.
>
> My question is: how much do we need to keep Ganglia integration in Spark
> 3.x? I think it does have some users. We can keep it as is and hope it
> works out in JDK 11, or consider dropping this module.
>
>
> -- Forwarded message -
> From: Apache Spark QA 
> Date: Wed, Oct 30, 2019 at 6:56 PM
> Subject: Re: [apache/spark] [SPARK-29674][CORE] Update dropwizard metrics
> to 4.1.x for JDK 9+ (#26332)
> To: apache/spark 
> Cc: Sean Owen , Assign 
>
>
> *Test build #112974 has started
> *
> for PR 26332 at commit aefde48
> 
> .
>
> —
> You are receiving this because you were assigned.
> Reply to this email directly, view it on GitHub
> ,
> or unsubscribe
> 
> .
>

Fwd: [apache/spark] [SPARK-29674][CORE] Update dropwizard metrics to 4.1.x for JDK 9+ (#26332)

[Sean Owen]

I wanted to raise this to dev@.

So, updating dropwizard metrics from 3.2.x to 4.x might be important for
JDK 11 support. Our tests pass as-is without this update. But we don't test
some elements of this metrics support, like Ganglia integration. And I have
heard reports that downstream custom usages of dropwizard 3.2.x doesn't
work on JDK 11.

The bad news is that the Ganglia integration doesn't exist anymore in 4.x.
And we have a whole custom module for that integration with Spark.

My question is: how much do we need to keep Ganglia integration in Spark
3.x? I think it does have some users. We can keep it as is and hope it
works out in JDK 11, or consider dropping this module.


-- Forwarded message -
From: Apache Spark QA 
Date: Wed, Oct 30, 2019 at 6:56 PM
Subject: Re: [apache/spark] [SPARK-29674][CORE] Update dropwizard metrics
to 4.1.x for JDK 9+ (#26332)
To: apache/spark 
Cc: Sean Owen , Assign 


*Test build #112974 has started
*
for PR 26332 at commit aefde48

.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
,
or unsubscribe

.

Re: [VOTE] SPARK 3.0.0-preview (RC1)

[Xingbo Jiang]

I was trying to avoid changing the version names and revert the changes on
master again. But you are right it might lead to confusions which release
script is used for RC2, I'll follow your advice and create a new RC2 tag.

Thanks!

Xingbo

On Wed, Oct 30, 2019 at 5:06 PM Dongjoon Hyun 
wrote:

> Hi, Xingbo.
>
> Currently, RC2 tag is pointing RC1 tag.
>
> https://github.com/apache/spark/tree/v3.0.0-preview-rc2
>
> Could you cut from the HEAD of master branch?
> Otherwise, nobody knows what release script you used for RC2.
>
> Bests,
> Dongjoon.
>
>
>
> On Wed, Oct 30, 2019 at 4:15 PM Xingbo Jiang 
> wrote:
>
>> Hi all,
>>
>> This RC fails because:
>> It fails to generate a PySpark release.
>>
>> I'll start RC2 soon.
>>
>> Thanks!
>>
>> Xingbo
>>
>>
>> On Wed, Oct 30, 2019 at 4:10 PM Xingbo Jiang 
>> wrote:
>>
>>> Thanks Sean, since we need to generate PySpark release with a different
>>> name, I would prefer fail RC1 and start another release candidate.
>>>
>>> Sean Owen  于2019年10月30日周三 下午4:00写道：
>>>
 I agree that we need a Pyspark release for this preview release. If
 it's a matter of producing it from the same tag, we can evaluate it
 within this same release candidate. Otherwise, just roll another
 release candidate.

 I was able to build it and pass all tests with JDK 8 and JDK 11
 (hadoop-3.2 profile, note) on Ubuntu, so this is otherwise looking
 good to me.

 On Tue, Oct 29, 2019 at 9:01 PM Xingbo Jiang 
 wrote:
 >
 > Please vote on releasing the following candidate as Apache Spark
 version 3.0.0-preview.
 >
 > The vote is open until November 2 PST and passes if a majority +1 PMC
 votes are cast, with
 > a minimum of 3 +1 votes.
 >
 > [ ] +1 Release this package as Apache Spark 3.0.0-preview
 > [ ] -1 Do not release this package because ...
 >
 > To learn more about Apache Spark, please see http://spark.apache.org/
 >
 > The tag to be voted on is v3.0.0-preview-rc1 (commit
 5eddbb5f1d9789696927f435c55df887e50a1389):
 > https://github.com/apache/spark/tree/v3.0.0-preview-rc1
 >
 > The release files, including signatures, digests, etc. can be found
 at:
 > https://dist.apache.org/repos/dist/dev/spark/v3.0.0-preview-rc1-bin/
 >
 > Signatures used for Spark RCs can be found in this file:
 > https://dist.apache.org/repos/dist/dev/spark/KEYS
 >
 > The staging repository for this release can be found at:
 >
 https://repository.apache.org/content/repositories/orgapachespark-1334/
 >
 > The documentation corresponding to this release can be found at:
 > https://dist.apache.org/repos/dist/dev/spark/v3.0.0-preview-rc1-docs/
 >
 > The list of bug fixes going into 3.0.0 can be found at the following
 URL:
 > https://issues.apache.org/jira/projects/SPARK/versions/12339177
 >
 > FAQ
 >
 > =
 > How can I help test this release?
 > =
 >
 > If you are a Spark user, you can help us test this release by taking
 > an existing Spark workload and running on this release candidate, then
 > reporting any regressions.
 >
 > If you're working in PySpark you can set up a virtual env and install
 > the current RC and see if anything important breaks, in the Java/Scala
 > you can add the staging repository to your projects resolvers and test
 > with the RC (make sure to clean up the artifact cache before/after so
 > you don't end up building with a out of date RC going forward).
 >
 > ===
 > What should happen to JIRA tickets still targeting 3.0.0?
 > ===
 >
 > The current list of open tickets targeted at 3.0.0 can be found at:
 > https://issues.apache.org/jira/projects/SPARK and search for "Target
 Version/s" = 3.0.0
 >
 > Committers should look at those and triage. Extremely important bug
 > fixes, documentation, and API tweaks that impact compatibility should
 > be worked on immediately. Everything else please retarget to an
 > appropriate release.
 >
 > ==
 > But my bug isn't fixed?
 > ==
 >
 > In order to make timely releases, we will typically not hold the
 > release unless the bug in question is a regression from the previous
 > release. That being said, if there is something which is a regression
 > that has not been correctly targeted please ping me or a committer to
 > help target the issue.

>>>

Re: [VOTE] SPARK 3.0.0-preview (RC1)

[Dongjoon Hyun]

Hi, Xingbo.

Currently, RC2 tag is pointing RC1 tag.

https://github.com/apache/spark/tree/v3.0.0-preview-rc2

Could you cut from the HEAD of master branch?
Otherwise, nobody knows what release script you used for RC2.

Bests,
Dongjoon.



On Wed, Oct 30, 2019 at 4:15 PM Xingbo Jiang  wrote:

> Hi all,
>
> This RC fails because:
> It fails to generate a PySpark release.
>
> I'll start RC2 soon.
>
> Thanks!
>
> Xingbo
>
>
> On Wed, Oct 30, 2019 at 4:10 PM Xingbo Jiang 
> wrote:
>
>> Thanks Sean, since we need to generate PySpark release with a different
>> name, I would prefer fail RC1 and start another release candidate.
>>
>> Sean Owen  于2019年10月30日周三 下午4:00写道：
>>
>>> I agree that we need a Pyspark release for this preview release. If
>>> it's a matter of producing it from the same tag, we can evaluate it
>>> within this same release candidate. Otherwise, just roll another
>>> release candidate.
>>>
>>> I was able to build it and pass all tests with JDK 8 and JDK 11
>>> (hadoop-3.2 profile, note) on Ubuntu, so this is otherwise looking
>>> good to me.
>>>
>>> On Tue, Oct 29, 2019 at 9:01 PM Xingbo Jiang 
>>> wrote:
>>> >
>>> > Please vote on releasing the following candidate as Apache Spark
>>> version 3.0.0-preview.
>>> >
>>> > The vote is open until November 2 PST and passes if a majority +1 PMC
>>> votes are cast, with
>>> > a minimum of 3 +1 votes.
>>> >
>>> > [ ] +1 Release this package as Apache Spark 3.0.0-preview
>>> > [ ] -1 Do not release this package because ...
>>> >
>>> > To learn more about Apache Spark, please see http://spark.apache.org/
>>> >
>>> > The tag to be voted on is v3.0.0-preview-rc1 (commit
>>> 5eddbb5f1d9789696927f435c55df887e50a1389):
>>> > https://github.com/apache/spark/tree/v3.0.0-preview-rc1
>>> >
>>> > The release files, including signatures, digests, etc. can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v3.0.0-preview-rc1-bin/
>>> >
>>> > Signatures used for Spark RCs can be found in this file:
>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>> >
>>> > The staging repository for this release can be found at:
>>> >
>>> https://repository.apache.org/content/repositories/orgapachespark-1334/
>>> >
>>> > The documentation corresponding to this release can be found at:
>>> > https://dist.apache.org/repos/dist/dev/spark/v3.0.0-preview-rc1-docs/
>>> >
>>> > The list of bug fixes going into 3.0.0 can be found at the following
>>> URL:
>>> > https://issues.apache.org/jira/projects/SPARK/versions/12339177
>>> >
>>> > FAQ
>>> >
>>> > =
>>> > How can I help test this release?
>>> > =
>>> >
>>> > If you are a Spark user, you can help us test this release by taking
>>> > an existing Spark workload and running on this release candidate, then
>>> > reporting any regressions.
>>> >
>>> > If you're working in PySpark you can set up a virtual env and install
>>> > the current RC and see if anything important breaks, in the Java/Scala
>>> > you can add the staging repository to your projects resolvers and test
>>> > with the RC (make sure to clean up the artifact cache before/after so
>>> > you don't end up building with a out of date RC going forward).
>>> >
>>> > ===
>>> > What should happen to JIRA tickets still targeting 3.0.0?
>>> > ===
>>> >
>>> > The current list of open tickets targeted at 3.0.0 can be found at:
>>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>>> Version/s" = 3.0.0
>>> >
>>> > Committers should look at those and triage. Extremely important bug
>>> > fixes, documentation, and API tweaks that impact compatibility should
>>> > be worked on immediately. Everything else please retarget to an
>>> > appropriate release.
>>> >
>>> > ==
>>> > But my bug isn't fixed?
>>> > ==
>>> >
>>> > In order to make timely releases, we will typically not hold the
>>> > release unless the bug in question is a regression from the previous
>>> > release. That being said, if there is something which is a regression
>>> > that has not been correctly targeted please ping me or a committer to
>>> > help target the issue.
>>>
>>

Re: [VOTE] SPARK 3.0.0-preview (RC1)

[Xingbo Jiang]

Hi all,

This RC fails because:
It fails to generate a PySpark release.

I'll start RC2 soon.

Thanks!

Xingbo


On Wed, Oct 30, 2019 at 4:10 PM Xingbo Jiang  wrote:

> Thanks Sean, since we need to generate PySpark release with a different
> name, I would prefer fail RC1 and start another release candidate.
>
> Sean Owen  于2019年10月30日周三 下午4:00写道：
>
>> I agree that we need a Pyspark release for this preview release. If
>> it's a matter of producing it from the same tag, we can evaluate it
>> within this same release candidate. Otherwise, just roll another
>> release candidate.
>>
>> I was able to build it and pass all tests with JDK 8 and JDK 11
>> (hadoop-3.2 profile, note) on Ubuntu, so this is otherwise looking
>> good to me.
>>
>> On Tue, Oct 29, 2019 at 9:01 PM Xingbo Jiang 
>> wrote:
>> >
>> > Please vote on releasing the following candidate as Apache Spark
>> version 3.0.0-preview.
>> >
>> > The vote is open until November 2 PST and passes if a majority +1 PMC
>> votes are cast, with
>> > a minimum of 3 +1 votes.
>> >
>> > [ ] +1 Release this package as Apache Spark 3.0.0-preview
>> > [ ] -1 Do not release this package because ...
>> >
>> > To learn more about Apache Spark, please see http://spark.apache.org/
>> >
>> > The tag to be voted on is v3.0.0-preview-rc1 (commit
>> 5eddbb5f1d9789696927f435c55df887e50a1389):
>> > https://github.com/apache/spark/tree/v3.0.0-preview-rc1
>> >
>> > The release files, including signatures, digests, etc. can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v3.0.0-preview-rc1-bin/
>> >
>> > Signatures used for Spark RCs can be found in this file:
>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>> >
>> > The staging repository for this release can be found at:
>> > https://repository.apache.org/content/repositories/orgapachespark-1334/
>> >
>> > The documentation corresponding to this release can be found at:
>> > https://dist.apache.org/repos/dist/dev/spark/v3.0.0-preview-rc1-docs/
>> >
>> > The list of bug fixes going into 3.0.0 can be found at the following
>> URL:
>> > https://issues.apache.org/jira/projects/SPARK/versions/12339177
>> >
>> > FAQ
>> >
>> > =
>> > How can I help test this release?
>> > =
>> >
>> > If you are a Spark user, you can help us test this release by taking
>> > an existing Spark workload and running on this release candidate, then
>> > reporting any regressions.
>> >
>> > If you're working in PySpark you can set up a virtual env and install
>> > the current RC and see if anything important breaks, in the Java/Scala
>> > you can add the staging repository to your projects resolvers and test
>> > with the RC (make sure to clean up the artifact cache before/after so
>> > you don't end up building with a out of date RC going forward).
>> >
>> > ===
>> > What should happen to JIRA tickets still targeting 3.0.0?
>> > ===
>> >
>> > The current list of open tickets targeted at 3.0.0 can be found at:
>> > https://issues.apache.org/jira/projects/SPARK and search for "Target
>> Version/s" = 3.0.0
>> >
>> > Committers should look at those and triage. Extremely important bug
>> > fixes, documentation, and API tweaks that impact compatibility should
>> > be worked on immediately. Everything else please retarget to an
>> > appropriate release.
>> >
>> > ==
>> > But my bug isn't fixed?
>> > ==
>> >
>> > In order to make timely releases, we will typically not hold the
>> > release unless the bug in question is a regression from the previous
>> > release. That being said, if there is something which is a regression
>> > that has not been correctly targeted please ping me or a committer to
>> > help target the issue.
>>
>

Re: [VOTE] SPARK 3.0.0-preview (RC1)

[Sean Owen]

I agree that we need a Pyspark release for this preview release. If
it's a matter of producing it from the same tag, we can evaluate it
within this same release candidate. Otherwise, just roll another
release candidate.

I was able to build it and pass all tests with JDK 8 and JDK 11
(hadoop-3.2 profile, note) on Ubuntu, so this is otherwise looking
good to me.

On Tue, Oct 29, 2019 at 9:01 PM Xingbo Jiang  wrote:
>
> Please vote on releasing the following candidate as Apache Spark version 
> 3.0.0-preview.
>
> The vote is open until November 2 PST and passes if a majority +1 PMC votes 
> are cast, with
> a minimum of 3 +1 votes.
>
> [ ] +1 Release this package as Apache Spark 3.0.0-preview
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is v3.0.0-preview-rc1 (commit 
> 5eddbb5f1d9789696927f435c55df887e50a1389):
> https://github.com/apache/spark/tree/v3.0.0-preview-rc1
>
> The release files, including signatures, digests, etc. can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.0.0-preview-rc1-bin/
>
> Signatures used for Spark RCs can be found in this file:
> https://dist.apache.org/repos/dist/dev/spark/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1334/
>
> The documentation corresponding to this release can be found at:
> https://dist.apache.org/repos/dist/dev/spark/v3.0.0-preview-rc1-docs/
>
> The list of bug fixes going into 3.0.0 can be found at the following URL:
> https://issues.apache.org/jira/projects/SPARK/versions/12339177
>
> FAQ
>
> =
> How can I help test this release?
> =
>
> If you are a Spark user, you can help us test this release by taking
> an existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> If you're working in PySpark you can set up a virtual env and install
> the current RC and see if anything important breaks, in the Java/Scala
> you can add the staging repository to your projects resolvers and test
> with the RC (make sure to clean up the artifact cache before/after so
> you don't end up building with a out of date RC going forward).
>
> ===
> What should happen to JIRA tickets still targeting 3.0.0?
> ===
>
> The current list of open tickets targeted at 3.0.0 can be found at:
> https://issues.apache.org/jira/projects/SPARK and search for "Target 
> Version/s" = 3.0.0
>
> Committers should look at those and triage. Extremely important bug
> fixes, documentation, and API tweaks that impact compatibility should
> be worked on immediately. Everything else please retarget to an
> appropriate release.
>
> ==
> But my bug isn't fixed?
> ==
>
> In order to make timely releases, we will typically not hold the
> release unless the bug in question is a regression from the previous
> release. That being said, if there is something which is a regression
> that has not been correctly targeted please ping me or a committer to
> help target the issue.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] SPARK 3.0.0-preview (RC1)

[Xingbo Jiang]

Thanks Sean, since we need to generate PySpark release with a different
name, I would prefer fail RC1 and start another release candidate.

Sean Owen  于2019年10月30日周三 下午4:00写道：

> I agree that we need a Pyspark release for this preview release. If
> it's a matter of producing it from the same tag, we can evaluate it
> within this same release candidate. Otherwise, just roll another
> release candidate.
>
> I was able to build it and pass all tests with JDK 8 and JDK 11
> (hadoop-3.2 profile, note) on Ubuntu, so this is otherwise looking
> good to me.
>
> On Tue, Oct 29, 2019 at 9:01 PM Xingbo Jiang 
> wrote:
> >
> > Please vote on releasing the following candidate as Apache Spark version
> 3.0.0-preview.
> >
> > The vote is open until November 2 PST and passes if a majority +1 PMC
> votes are cast, with
> > a minimum of 3 +1 votes.
> >
> > [ ] +1 Release this package as Apache Spark 3.0.0-preview
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see http://spark.apache.org/
> >
> > The tag to be voted on is v3.0.0-preview-rc1 (commit
> 5eddbb5f1d9789696927f435c55df887e50a1389):
> > https://github.com/apache/spark/tree/v3.0.0-preview-rc1
> >
> > The release files, including signatures, digests, etc. can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v3.0.0-preview-rc1-bin/
> >
> > Signatures used for Spark RCs can be found in this file:
> > https://dist.apache.org/repos/dist/dev/spark/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1334/
> >
> > The documentation corresponding to this release can be found at:
> > https://dist.apache.org/repos/dist/dev/spark/v3.0.0-preview-rc1-docs/
> >
> > The list of bug fixes going into 3.0.0 can be found at the following URL:
> > https://issues.apache.org/jira/projects/SPARK/versions/12339177
> >
> > FAQ
> >
> > =
> > How can I help test this release?
> > =
> >
> > If you are a Spark user, you can help us test this release by taking
> > an existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > If you're working in PySpark you can set up a virtual env and install
> > the current RC and see if anything important breaks, in the Java/Scala
> > you can add the staging repository to your projects resolvers and test
> > with the RC (make sure to clean up the artifact cache before/after so
> > you don't end up building with a out of date RC going forward).
> >
> > ===
> > What should happen to JIRA tickets still targeting 3.0.0?
> > ===
> >
> > The current list of open tickets targeted at 3.0.0 can be found at:
> > https://issues.apache.org/jira/projects/SPARK and search for "Target
> Version/s" = 3.0.0
> >
> > Committers should look at those and triage. Extremely important bug
> > fixes, documentation, and API tweaks that impact compatibility should
> > be worked on immediately. Everything else please retarget to an
> > appropriate release.
> >
> > ==
> > But my bug isn't fixed?
> > ==
> >
> > In order to make timely releases, we will typically not hold the
> > release unless the bug in question is a regression from the previous
> > release. That being said, if there is something which is a regression
> > that has not been correctly targeted please ping me or a committer to
> > help target the issue.
>

Re: Packages to release in 3.0.0-preview

[Sean Owen]

I don't agree with this take. The bottleneck is pretty much not Spark
-- it is all of its dependencies, and there are unfortunately a lot.
For example, Chill (among other things) doesn't support 2.13 yet. I
don't think 2.13 is that 'mainstream' yet. We are not close to Scala
2.13 support, so it won't be in 3.0, but, I can tell you I've already
made almost all of the Spark-side changes for it.

Keep in mind too that using Scala 2.13 for users means that all of
_their_ dependencies have to support 2.13.

Please instead look at the JIRAs for 2.13 support and encourage
dependencies to update.
Or, frankly, maybe Scala should reconsider the mutual incompatibility
between minor releases. These are basically major releases, and
indeed, it causes exactly this kind of headache.

On Wed, Oct 30, 2019 at 5:36 PM antonkulaga  wrote:
>
> Why not trying the current Scala (2.13)? Spark has always been one (sometimes
> - two) Scala versions away from the whole Scala ecosystem and it has always
> been a big pain point for everybody. I understand that in the past you could
> not switch because of compatibility issues, but 3.x is a major version
> update and you can break things, maybe you can finally consider to use the
> current Scala?
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Packages to release in 3.0.0-preview

[Xingbo Jiang]

scala 2.13 support is tracked by
https://issues.apache.org/jira/browse/SPARK-25075 , at the current time
there are still major issues remaining, thus we don't include scala 2.13
support in the 3.0.0-preview release.
If the task is finished before the code freeze of Spark 3.0.0, then it's
still possible to release Spark 3.0.0 with scala 2.13 packages.

Cheers,

Xingbo

antonkulaga  于2019年10月30日周三 下午3:36写道：

> Why not trying the current Scala (2.13)? Spark has always been one
> (sometimes
> - two) Scala versions away from the whole Scala ecosystem and it has always
> been a big pain point for everybody. I understand that in the past you
> could
> not switch because of compatibility issues, but 3.x is a major version
> update and you can break things, maybe you can finally consider to use the
> current Scala?
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: Packages to release in 3.0.0-preview

[antonkulaga]

Why not trying the current Scala (2.13)? Spark has always been one (sometimes
- two) Scala versions away from the whole Scala ecosystem and it has always
been a big pain point for everybody. I understand that in the past you could
not switch because of compatibility issues, but 3.x is a major version
update and you can break things, maybe you can finally consider to use the
current Scala?



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

[Shane Knapp]

sure.  that shouldn't be too hard, but we've historically given very little
support to it.

On Wed, Oct 30, 2019 at 2:31 PM Maciej Szymkiewicz 
wrote:

> Could we upgrade to PyPy3.6 v7.2.0?
> On 10/30/19 9:45 PM, Shane Knapp wrote:
>
> one quick thing:  we currently test against python2.7, 3.6 *and* pypy2.5.1
> (python2.7).
>
> what are our plans for pypy?
>
>
> On Wed, Oct 30, 2019 at 12:26 PM Dongjoon Hyun 
> wrote:
>
>> Thank you all. I made a PR for that.
>>
>> https://github.com/apache/spark/pull/26326
>>
>> On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro 
>> wrote:
>>
>>> +1, too.
>>>
>>> On Tue, Oct 29, 2019 at 4:16 PM Holden Karau 
>>> wrote:
>>>
 +1 to deprecating but not yet removing support for 3.6

 On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp 
 wrote:

> +1 to testing the absolute minimum number of python variants as
> possible.  ;)
>
> On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon 
> wrote:
>
>> +1 from me as well.
>>
>> 2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng 님이 작성:
>>
>>> +1. And we should start testing 3.7 and maybe 3.8 in Jenkins.
>>>
>>> On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <
>>> dongjoon.h...@gmail.com> wrote:
>>>
 Thank you for starting the thread.

 In addition to that, we currently are testing Python 3.6 only in
 Apache Spark Jenkins environment.

 Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1
 will start next January
 (https://spark.apache.org/versioning-policy.html), I'm +1 for the
 deprecation (Python < 3.6) at Apache Spark 3.0.0.

 It's just a deprecation to prepare the next-step development cycle.
 Bests,
 Dongjoon.


 On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <
 mszymkiew...@gmail.com> wrote:

> Hi everyone,
>
> While deprecation of Python 2 in 3.0.0 has been announced
> ,
> there is no clear statement about specific continuing support of 
> different
> Python 3 version.
>
> Specifically:
>
>- Python 3.4 has been retired this year.
>- Python 3.5 is already in the "security fixes only" mode and
>should be retired in the middle of 2020.
>
> Continued support of these two blocks adoption of many new Python
> features (PEP 468)  and it is hard to justify beyond 2020.
>
> Should these two be deprecated in 3.0.0 as well?
>
> --
> Best regards,
> Maciej
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
 --
 Twitter: https://twitter.com/holdenkarau
 Books (Learning Spark, High Performance Spark, etc.):
 https://amzn.to/2MaRAG9  
 YouTube Live Streams: https://www.youtube.com/user/holdenkarau

>>>
>>>
>>> --
>>> ---
>>> Takeshi Yamamuro
>>>
>>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
> --
> Best regards,
> Maciej
>
>

-- 
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

[Maciej Szymkiewicz]

Could we upgrade to PyPy3.6 v7.2.0?

On 10/30/19 9:45 PM, Shane Knapp wrote:
> one quick thing:  we currently test against python2.7, 3.6 *and*
> pypy2.5.1 (python2.7).
>
> what are our plans for pypy?
>
>
> On Wed, Oct 30, 2019 at 12:26 PM Dongjoon Hyun
> mailto:dongjoon.h...@gmail.com>> wrote:
>
> Thank you all. I made a PR for that.
>
> https://github.com/apache/spark/pull/26326
>
> On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro
> mailto:linguin@gmail.com>> wrote:
>
> +1, too.
>
> On Tue, Oct 29, 2019 at 4:16 PM Holden Karau
> mailto:hol...@pigscanfly.ca>> wrote:
>
> +1 to deprecating but not yet removing support for 3.6
>
> On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp
> mailto:skn...@berkeley.edu>> wrote:
>
> +1 to testing the absolute minimum number of python
> variants as possible.  ;)
>
> On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon
> mailto:gurwls...@gmail.com>> wrote:
>
> +1 from me as well.
>
> 2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng
>  >님이 작성:
>
> +1. And we should start testing 3.7 and maybe
> 3.8 in Jenkins.
>
> On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun
>  > wrote:
>
> Thank you for starting the thread.
>
> In addition to that, we currently are
> testing Python 3.6 only in Apache Spark
> Jenkins environment.
>
> Given that Python 3.8 is already out and
> Apache Spark 3.0.0 RC1 will start next January
> (https://spark.apache.org/versioning-policy.html),
> I'm +1 for the deprecation (Python < 3.6)
> at Apache Spark 3.0.0.
>
> It's just a deprecation to prepare the
> next-step development cycle.
> Bests,
> Dongjoon.
>
>
> On Thu, Oct 24, 2019 at 1:10 AM Maciej
> Szymkiewicz  > wrote:
>
> Hi everyone,
>
> While deprecation of Python 2 in 3.0.0
> has been announced
> 
> ,
> there is no clear statement about
> specific continuing support of
> different Python 3 version.
>
> Specifically:
>
>   * Python 3.4 has been retired this year.
>   * Python 3.5 is already in the
> "security fixes only" mode and
> should be retired in the middle of
> 2020.
>
> Continued support of these two blocks
> adoption of many new Python features
> (PEP 468)  and it is hard to justify
> beyond 2020.
>
> Should these two be deprecated in
> 3.0.0 as well?
>
> -- 
> Best regards,
> Maciej
>
>
>
> -- 
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
> -- 
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark,
> etc.): https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
>
>
> -- 
> ---
> Takeshi Yamamuro
>
>
>
> -- 
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu

-- 
Best regards,
Maciej

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

[Shane Knapp]

also, here's my PR for dropping 2.7 tests:
https://github.com/apache/spark/pull/26330

On Wed, Oct 30, 2019 at 1:45 PM Shane Knapp  wrote:

> one quick thing:  we currently test against python2.7, 3.6 *and* pypy2.5.1
> (python2.7).
>
> what are our plans for pypy?
>
>
> On Wed, Oct 30, 2019 at 12:26 PM Dongjoon Hyun 
> wrote:
>
>> Thank you all. I made a PR for that.
>>
>> https://github.com/apache/spark/pull/26326
>>
>> On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro 
>> wrote:
>>
>>> +1, too.
>>>
>>> On Tue, Oct 29, 2019 at 4:16 PM Holden Karau 
>>> wrote:
>>>
 +1 to deprecating but not yet removing support for 3.6

 On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp 
 wrote:

> +1 to testing the absolute minimum number of python variants as
> possible.  ;)
>
> On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon 
> wrote:
>
>> +1 from me as well.
>>
>> 2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng 님이 작성:
>>
>>> +1. And we should start testing 3.7 and maybe 3.8 in Jenkins.
>>>
>>> On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <
>>> dongjoon.h...@gmail.com> wrote:
>>>
 Thank you for starting the thread.

 In addition to that, we currently are testing Python 3.6 only in
 Apache Spark Jenkins environment.

 Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1
 will start next January
 (https://spark.apache.org/versioning-policy.html), I'm +1 for the
 deprecation (Python < 3.6) at Apache Spark 3.0.0.

 It's just a deprecation to prepare the next-step development cycle.

 Bests,
 Dongjoon.


 On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <
 mszymkiew...@gmail.com> wrote:

> Hi everyone,
>
> While deprecation of Python 2 in 3.0.0 has been announced
> ,
> there is no clear statement about specific continuing support of 
> different
> Python 3 version.
>
> Specifically:
>
>- Python 3.4 has been retired this year.
>- Python 3.5 is already in the "security fixes only" mode and
>should be retired in the middle of 2020.
>
> Continued support of these two blocks adoption of many new Python
> features (PEP 468)  and it is hard to justify beyond 2020.
>
> Should these two be deprecated in 3.0.0 as well?
>
> --
> Best regards,
> Maciej
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
 --
 Twitter: https://twitter.com/holdenkarau
 Books (Learning Spark, High Performance Spark, etc.):
 https://amzn.to/2MaRAG9  
 YouTube Live Streams: https://www.youtube.com/user/holdenkarau

>>>
>>>
>>> --
>>> ---
>>> Takeshi Yamamuro
>>>
>>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>


-- 
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

[Shane Knapp]

one quick thing:  we currently test against python2.7, 3.6 *and* pypy2.5.1
(python2.7).

what are our plans for pypy?


On Wed, Oct 30, 2019 at 12:26 PM Dongjoon Hyun 
wrote:

> Thank you all. I made a PR for that.
>
> https://github.com/apache/spark/pull/26326
>
> On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro 
> wrote:
>
>> +1, too.
>>
>> On Tue, Oct 29, 2019 at 4:16 PM Holden Karau 
>> wrote:
>>
>>> +1 to deprecating but not yet removing support for 3.6
>>>
>>> On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp  wrote:
>>>
 +1 to testing the absolute minimum number of python variants as
 possible.  ;)

 On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon 
 wrote:

> +1 from me as well.
>
> 2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng 님이 작성:
>
>> +1. And we should start testing 3.7 and maybe 3.8 in Jenkins.
>>
>> On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <
>> dongjoon.h...@gmail.com> wrote:
>>
>>> Thank you for starting the thread.
>>>
>>> In addition to that, we currently are testing Python 3.6 only in
>>> Apache Spark Jenkins environment.
>>>
>>> Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will
>>> start next January
>>> (https://spark.apache.org/versioning-policy.html), I'm +1 for the
>>> deprecation (Python < 3.6) at Apache Spark 3.0.0.
>>>
>>> It's just a deprecation to prepare the next-step development cycle.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>>
>>> On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <
>>> mszymkiew...@gmail.com> wrote:
>>>
 Hi everyone,

 While deprecation of Python 2 in 3.0.0 has been announced
 ,
 there is no clear statement about specific continuing support of 
 different
 Python 3 version.

 Specifically:

- Python 3.4 has been retired this year.
- Python 3.5 is already in the "security fixes only" mode and
should be retired in the middle of 2020.

 Continued support of these two blocks adoption of many new Python
 features (PEP 468)  and it is hard to justify beyond 2020.

 Should these two be deprecated in 3.0.0 as well?

 --
 Best regards,
 Maciej



 --
 Shane Knapp
 UC Berkeley EECS Research / RISELab Staff Technical Lead
 https://rise.cs.berkeley.edu

>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>>
>> --
>> ---
>> Takeshi Yamamuro
>>
>

-- 
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

[Dongjoon Hyun]

Thank you all. I made a PR for that.

https://github.com/apache/spark/pull/26326

On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro 
wrote:

> +1, too.
>
> On Tue, Oct 29, 2019 at 4:16 PM Holden Karau  wrote:
>
>> +1 to deprecating but not yet removing support for 3.6
>>
>> On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp  wrote:
>>
>>> +1 to testing the absolute minimum number of python variants as
>>> possible.  ;)
>>>
>>> On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon 
>>> wrote:
>>>
 +1 from me as well.

 2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng 님이 작성:

> +1. And we should start testing 3.7 and maybe 3.8 in Jenkins.
>
> On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun 
> wrote:
>
>> Thank you for starting the thread.
>>
>> In addition to that, we currently are testing Python 3.6 only in
>> Apache Spark Jenkins environment.
>>
>> Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will
>> start next January
>> (https://spark.apache.org/versioning-policy.html), I'm +1 for the
>> deprecation (Python < 3.6) at Apache Spark 3.0.0.
>>
>> It's just a deprecation to prepare the next-step development cycle.
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <
>> mszymkiew...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> While deprecation of Python 2 in 3.0.0 has been announced
>>> ,
>>> there is no clear statement about specific continuing support of 
>>> different
>>> Python 3 version.
>>>
>>> Specifically:
>>>
>>>- Python 3.4 has been retired this year.
>>>- Python 3.5 is already in the "security fixes only" mode and
>>>should be retired in the middle of 2020.
>>>
>>> Continued support of these two blocks adoption of many new Python
>>> features (PEP 468)  and it is hard to justify beyond 2020.
>>>
>>> Should these two be deprecated in 3.0.0 as well?
>>>
>>> --
>>> Best regards,
>>> Maciej
>>>
>>>
>>>
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
>
> --
> ---
> Takeshi Yamamuro
>