Re: JDK version support policy?

[Jungtaek Lim]

+1 to drop Java 8 but +1 to set the lowest support version to Java 11.

Considering the phase for only security updates, 11 LTS would not be EOLed
in very long time. Unless that’s coupled with other deps which require
bumping JDK version (hope someone can bring up lists), it doesn’t seem to
buy much. And given the strong backward compatibility JDK provides, that’s
less likely.

Purely from the project’s source code view, does anyone know how much
benefits we can leverage for picking up 17 rather than 11? I lost the
track, but some of their proposals are more likely catching up with other
languages, which don’t make us be happy since Scala provides them for years.

2023년 6월 8일 (목) 오전 2:35, Sean Owen 님이 작성:

> I also generally perceive that, after Java 9, there is much less breaking
> change. So working on Java 11 probably means it works on 20, or can be
> easily made to without pain. Like I think the tweaks for Java 17 were quite
> small.
>
> Targeting Java >11 excludes Java 11 users and probably wouldn't buy much.
> Keeping the support probably doesn't interfere with working on much newer
> JVMs either.
>
> On Wed, Jun 7, 2023, 12:29 PM Holden Karau  wrote:
>
>> So JDK 11 is still supported in open JDK until 2026, I'm not sure if
>> we're going to see enough folks moving to JRE17 by the Spark 4 release
>> unless we have a strong benefit from dropping 11 support I'd be inclined to
>> keep it.
>>
>> On Tue, Jun 6, 2023 at 9:08 PM Dongjoon Hyun  wrote:
>>
>>> I'm also +1 on dropping both Java 8 and 11 in Apache Spark 4.0, too.
>>>
>>> Dongjoon.
>>>
>>> On 2023/06/07 02:42:19 yangjie01 wrote:
>>> > +1 on dropping Java 8 in Spark 4.0, and I even hope Spark 4.0 can only
>>> support Java 17 and the upcoming Java 21.
>>> >
>>> > 发件人: Denny Lee 
>>> > 日期: 2023年6月7日 星期三 07:10
>>> > 收件人: Sean Owen 
>>> > 抄送: David Li , "dev@spark.apache.org" <
>>> dev@spark.apache.org>
>>> > 主题: Re: JDK version support policy?
>>> >
>>> > +1 on dropping Java 8 in Spark 4.0, saying this as a fan of the
>>> fast-paced (positive) updates to Arrow, eh?!
>>> >
>>> > On Tue, Jun 6, 2023 at 4:02 PM Sean Owen >> sro...@gmail.com>> wrote:
>>> > I haven't followed this discussion closely, but I think we
>>> could/should drop Java 8 in Spark 4.0, which is up next after 3.5?
>>> >
>>> > On Tue, Jun 6, 2023 at 2:44 PM David Li >> lidav...@apache.org>> wrote:
>>> > Hello Spark developers,
>>> >
>>> > I'm from the Apache Arrow project. We've discussed Java version
>>> support [1], and crucially, whether to continue supporting Java 8 or not.
>>> As Spark is a big user of Arrow in Java, I was curious what Spark's policy
>>> here was.
>>> >
>>> > If Spark intends to stay on Java 8, for instance, we may also want to
>>> stay on Java 8 or otherwise provide some supported version of Arrow for
>>> Java 8.
>>> >
>>> > We've seen dependencies dropping or planning to drop support. gRPC may
>>> drop Java 8 at any time [2], possibly this September [3], which may affect
>>> Spark (due to Spark Connect). And today we saw that Arrow had issues
>>> running tests with Mockito on Java 20, but we couldn't update Mockito since
>>> it had dropped Java 8 support. (We pinned the JDK version in that CI
>>> pipeline for now.)
>>> >
>>> > So at least, I am curious if Arrow could start the long process of
>>> migrating Java versions without impacting Spark, or if we should continue
>>> to cooperate. Arrow Java doesn't see quite so much activity these days, so
>>> it's not quite critical, but it's possible that these dependency issues
>>> will start to affect us more soon. And looking forward, Java is working on
>>> APIs that should also allow us to ditch the --add-opens flag requirement
>>> too.
>>> >
>>> > [1]: https://lists.apache.org/thread/phpgpydtt3yrgnncdyv4qdq1gf02s0yj<
>>> https://mailshield.baidu.com/check?q=Nz%2bGj2hdKguk92URjA7sg0PfbSN%2fXUIMgrHTmW45gOOKEr3Shre45B7TRzhEpb%2baVsnyuRL%2fl%2f0cu7IVGHunSGDVnxM%3d
>>> >
>>> > [2]:
>>> https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md<
>>> https://mailshield.baidu.com/check?q=s89S3eo8GCJkV7Mpx7aG1SXId7uCRYGjQMA6DeLuX9duS86LhIODZMJfeFdGMWdFzJ8S7minyHoC7mCrzHagbJXCXYTBH%2fpZBpfTbw%3d%3d
>>> >
>>> > [3]: https://github.com/grpc/grpc-java/issues/9386<
>>> https://mailshield.baidu.com/check?q=R0HtWZIkY5eIxpz8jtqHLzd0ugNbcaXIKW2LbUUxpIn0t9Y9yAhuHPuZ4buryfNwRnnJTA%3d%3d
>>> >
>>> >
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

Re: ASF policy violation and Scala version issues

[Grisha Weintraub]

Dongjoon,

I followed the conversation, and in my opinion, your concern is totally
legit.
It just feels that the discussion is focused solely on Databricks, and as I
said above, the same issue occurs in other vendors as well.


On Wed, Jun 7, 2023 at 10:28 PM Dongjoon Hyun 
wrote:

> To Grisha, we are talking about what is the right way and how to comply
> with ASF legal advice which I shared in this thread from "legal-discuss@"
> mailing thread.
>
> https://lists.apache.org/thread/mzhggd0rpz8t4d7vdsbhkp38mvd3lty4
>  (legal-discuss@)
> https://www.apache.org/foundation/marks/downstream.html#source (ASF
> Website)
>
> Dongjoon
>
>
> On Wed, Jun 7, 2023 at 12:16 PM Grisha Weintraub <
> grisha.weintr...@gmail.com> wrote:
>
>> Yes, in Spark UI you have it as "3.1.2-amazon", but when you create a
>> cluster it's just Spark 3.1.2.
>>
>> On Wed, Jun 7, 2023 at 10:05 PM Nan Zhu  wrote:
>>
>>>
>>>  for EMR, I think they show 3.1.2-amazon in Spark UI, no?
>>>
>>>
>>> On Wed, Jun 7, 2023 at 11:30 Grisha Weintraub <
>>> grisha.weintr...@gmail.com> wrote:
>>>
 Hi,

 I am not taking sides here, but just for fairness, I think it should be
 noted that AWS EMR does exactly the same thing.
 We choose the EMR version (e.g., 6.4.0) and it has an associated Spark
 version (e.g., 3.1.2).
 The Spark version here is not the original Apache version but AWS Spark
 distribution.

 On Wed, Jun 7, 2023 at 8:24 PM Dongjoon Hyun 
 wrote:

> I disagree with you in several ways.
>
> The following is not a *minor* change like the given examples
> (alterations to the start-up and shutdown scripts, configuration files,
> file layout etc.).
>
> > The change you cite meets the 4th point, minor change, made for
> integration reasons.
>
> The following is also wrong. There is no such point of state of Apache
> Spark 3.4.0 after 3.4.0 tag creation. Apache Spark community didn't allow
> Scala reverting patches in both `master` branch and `branch-3.4`.
>
> > There is no known technical objection; this was after all at one
> point the state of Apache Spark.
>
> Is the following your main point? So, you are selling a box "including
> Harry Potter by J. K. Rolling whose main character is Barry instead of
> Harry", but it's okay because you didn't sell the book itself? And, as a
> cloud-vendor, you borrowed the box instead of selling it like private
> libraries?
>
> > There is no standalone distribution of Apache Spark anywhere here.
>
> We are not asking a big thing. Why are you so reluctant to say you are
> not "Apache Spark 3.4.0" by simply saying "Apache Spark 3.4.0-databricks".
> What is the marketing reason here?
>
> Dongjoon.
>
>
> On Wed, Jun 7, 2023 at 9:27 AM Sean Owen  wrote:
>
>> Hi Dongjoon, I think this conversation is not advancing anymore. I
>> personally consider the matter closed unless you can find other support 
>> or
>> respond with more specifics. While this perhaps should be on private@,
>> I think it's not wrong as an instructive discussion on dev@.
>>
>> I don't believe you've made a clear argument about the problem, or
>> how it relates specifically to policy. Nevertheless I will show you my
>> logic.
>>
>> You are asserting that a vendor cannot call a product Apache Spark
>> 3.4.0 if it omits a patch updating a Scala maintenance version. This
>> difference has no known impact on usage, as far as I can tell.
>>
>> Let's see what policy requires:
>>
>> 1/ All source code changes must meet at least one of the acceptable
>> changes criteria set out below:
>> - The change has accepted by the relevant Apache project community
>> for inclusion in a future release. Note that the process used to accept
>> changes and how that acceptance is documented varies between projects.
>> - A change is a fix for an undisclosed security issue; and the fix is
>> not publicly disclosed as as security fix; and the Apache project has 
>> been
>> notified of the both issue and the proposed fix; and the PMC has rejected
>> neither the vulnerability report nor the proposed fix.
>> - A change is a fix for a bug; and the Apache project has been
>> notified of both the bug and the proposed fix; and the PMC has rejected
>> neither the bug report nor the proposed fix.
>> - Minor changes (e.g. alterations to the start-up and shutdown
>> scripts, configuration files, file layout etc.) to integrate with the
>> target platform providing the Apache project has not objected to those
>> changes.
>>
>> The change you cite meets the 4th point, minor change, made for
>> integration reasons. There is no known technical objection; this was 
>> after
>> all at one point the state of Apache Spark.
>>
>>
>> 2/ A version number must be 

Re: ASF policy violation and Scala version issues

[Dongjoon Hyun]

To Grisha, we are talking about what is the right way and how to comply
with ASF legal advice which I shared in this thread from "legal-discuss@"
mailing thread.

https://lists.apache.org/thread/mzhggd0rpz8t4d7vdsbhkp38mvd3lty4
 (legal-discuss@)
https://www.apache.org/foundation/marks/downstream.html#source (ASF Website)

Dongjoon


On Wed, Jun 7, 2023 at 12:16 PM Grisha Weintraub 
wrote:

> Yes, in Spark UI you have it as "3.1.2-amazon", but when you create a
> cluster it's just Spark 3.1.2.
>
> On Wed, Jun 7, 2023 at 10:05 PM Nan Zhu  wrote:
>
>>
>>  for EMR, I think they show 3.1.2-amazon in Spark UI, no?
>>
>>
>> On Wed, Jun 7, 2023 at 11:30 Grisha Weintraub 
>> wrote:
>>
>>> Hi,
>>>
>>> I am not taking sides here, but just for fairness, I think it should be
>>> noted that AWS EMR does exactly the same thing.
>>> We choose the EMR version (e.g., 6.4.0) and it has an associated Spark
>>> version (e.g., 3.1.2).
>>> The Spark version here is not the original Apache version but AWS Spark
>>> distribution.
>>>
>>> On Wed, Jun 7, 2023 at 8:24 PM Dongjoon Hyun 
>>> wrote:
>>>
 I disagree with you in several ways.

 The following is not a *minor* change like the given examples
 (alterations to the start-up and shutdown scripts, configuration files,
 file layout etc.).

 > The change you cite meets the 4th point, minor change, made for
 integration reasons.

 The following is also wrong. There is no such point of state of Apache
 Spark 3.4.0 after 3.4.0 tag creation. Apache Spark community didn't allow
 Scala reverting patches in both `master` branch and `branch-3.4`.

 > There is no known technical objection; this was after all at one
 point the state of Apache Spark.

 Is the following your main point? So, you are selling a box "including
 Harry Potter by J. K. Rolling whose main character is Barry instead of
 Harry", but it's okay because you didn't sell the book itself? And, as a
 cloud-vendor, you borrowed the box instead of selling it like private
 libraries?

 > There is no standalone distribution of Apache Spark anywhere here.

 We are not asking a big thing. Why are you so reluctant to say you are
 not "Apache Spark 3.4.0" by simply saying "Apache Spark 3.4.0-databricks".
 What is the marketing reason here?

 Dongjoon.


 On Wed, Jun 7, 2023 at 9:27 AM Sean Owen  wrote:

> Hi Dongjoon, I think this conversation is not advancing anymore. I
> personally consider the matter closed unless you can find other support or
> respond with more specifics. While this perhaps should be on private@,
> I think it's not wrong as an instructive discussion on dev@.
>
> I don't believe you've made a clear argument about the problem, or how
> it relates specifically to policy. Nevertheless I will show you my logic.
>
> You are asserting that a vendor cannot call a product Apache Spark
> 3.4.0 if it omits a patch updating a Scala maintenance version. This
> difference has no known impact on usage, as far as I can tell.
>
> Let's see what policy requires:
>
> 1/ All source code changes must meet at least one of the acceptable
> changes criteria set out below:
> - The change has accepted by the relevant Apache project community for
> inclusion in a future release. Note that the process used to accept 
> changes
> and how that acceptance is documented varies between projects.
> - A change is a fix for an undisclosed security issue; and the fix is
> not publicly disclosed as as security fix; and the Apache project has been
> notified of the both issue and the proposed fix; and the PMC has rejected
> neither the vulnerability report nor the proposed fix.
> - A change is a fix for a bug; and the Apache project has been
> notified of both the bug and the proposed fix; and the PMC has rejected
> neither the bug report nor the proposed fix.
> - Minor changes (e.g. alterations to the start-up and shutdown
> scripts, configuration files, file layout etc.) to integrate with the
> target platform providing the Apache project has not objected to those
> changes.
>
> The change you cite meets the 4th point, minor change, made for
> integration reasons. There is no known technical objection; this was after
> all at one point the state of Apache Spark.
>
>
> 2/ A version number must be used that both clearly differentiates it
> from an Apache Software Foundation release and clearly identifies the
> Apache Software Foundation version on which the software is based.
>
> Keep in mind the product here is not "Apache Spark", but the
> "Databricks Runtime 13.1 (including Apache Spark 3.4.0)". That is, there 
> is
> far more than a version number differentiating this product from Apache
> Spark. There is no standalone distribution of 

Re: ASF policy violation and Scala version issues

[Grisha Weintraub]

Yes, in Spark UI you have it as "3.1.2-amazon", but when you create a
cluster it's just Spark 3.1.2.

On Wed, Jun 7, 2023 at 10:05 PM Nan Zhu  wrote:

>
>  for EMR, I think they show 3.1.2-amazon in Spark UI, no?
>
>
> On Wed, Jun 7, 2023 at 11:30 Grisha Weintraub 
> wrote:
>
>> Hi,
>>
>> I am not taking sides here, but just for fairness, I think it should be
>> noted that AWS EMR does exactly the same thing.
>> We choose the EMR version (e.g., 6.4.0) and it has an associated Spark
>> version (e.g., 3.1.2).
>> The Spark version here is not the original Apache version but AWS Spark
>> distribution.
>>
>> On Wed, Jun 7, 2023 at 8:24 PM Dongjoon Hyun 
>> wrote:
>>
>>> I disagree with you in several ways.
>>>
>>> The following is not a *minor* change like the given examples
>>> (alterations to the start-up and shutdown scripts, configuration files,
>>> file layout etc.).
>>>
>>> > The change you cite meets the 4th point, minor change, made for
>>> integration reasons.
>>>
>>> The following is also wrong. There is no such point of state of Apache
>>> Spark 3.4.0 after 3.4.0 tag creation. Apache Spark community didn't allow
>>> Scala reverting patches in both `master` branch and `branch-3.4`.
>>>
>>> > There is no known technical objection; this was after all at one point
>>> the state of Apache Spark.
>>>
>>> Is the following your main point? So, you are selling a box "including
>>> Harry Potter by J. K. Rolling whose main character is Barry instead of
>>> Harry", but it's okay because you didn't sell the book itself? And, as a
>>> cloud-vendor, you borrowed the box instead of selling it like private
>>> libraries?
>>>
>>> > There is no standalone distribution of Apache Spark anywhere here.
>>>
>>> We are not asking a big thing. Why are you so reluctant to say you are
>>> not "Apache Spark 3.4.0" by simply saying "Apache Spark 3.4.0-databricks".
>>> What is the marketing reason here?
>>>
>>> Dongjoon.
>>>
>>>
>>> On Wed, Jun 7, 2023 at 9:27 AM Sean Owen  wrote:
>>>
 Hi Dongjoon, I think this conversation is not advancing anymore. I
 personally consider the matter closed unless you can find other support or
 respond with more specifics. While this perhaps should be on private@,
 I think it's not wrong as an instructive discussion on dev@.

 I don't believe you've made a clear argument about the problem, or how
 it relates specifically to policy. Nevertheless I will show you my logic.

 You are asserting that a vendor cannot call a product Apache Spark
 3.4.0 if it omits a patch updating a Scala maintenance version. This
 difference has no known impact on usage, as far as I can tell.

 Let's see what policy requires:

 1/ All source code changes must meet at least one of the acceptable
 changes criteria set out below:
 - The change has accepted by the relevant Apache project community for
 inclusion in a future release. Note that the process used to accept changes
 and how that acceptance is documented varies between projects.
 - A change is a fix for an undisclosed security issue; and the fix is
 not publicly disclosed as as security fix; and the Apache project has been
 notified of the both issue and the proposed fix; and the PMC has rejected
 neither the vulnerability report nor the proposed fix.
 - A change is a fix for a bug; and the Apache project has been notified
 of both the bug and the proposed fix; and the PMC has rejected neither the
 bug report nor the proposed fix.
 - Minor changes (e.g. alterations to the start-up and shutdown scripts,
 configuration files, file layout etc.) to integrate with the target
 platform providing the Apache project has not objected to those changes.

 The change you cite meets the 4th point, minor change, made for
 integration reasons. There is no known technical objection; this was after
 all at one point the state of Apache Spark.


 2/ A version number must be used that both clearly differentiates it
 from an Apache Software Foundation release and clearly identifies the
 Apache Software Foundation version on which the software is based.

 Keep in mind the product here is not "Apache Spark", but the
 "Databricks Runtime 13.1 (including Apache Spark 3.4.0)". That is, there is
 far more than a version number differentiating this product from Apache
 Spark. There is no standalone distribution of Apache Spark anywhere here. I
 believe that easily matches the intent.


 3/ The documentation must clearly identify the Apache Software
 Foundation version on which the software is based.

 Clearly, yes.


 4/ The end user expects that the distribution channel will back-port
 fixes. It is not necessary to back-port all fixes. Selection of fixes to
 back-port must be consistent with the update policy of that distribution
 channel.

 I think this is safe 

Re: ASF policy violation and Scala version issues

[Nan Zhu]

 for EMR, I think they show 3.1.2-amazon in Spark UI, no?


On Wed, Jun 7, 2023 at 11:30 Grisha Weintraub 
wrote:

> Hi,
>
> I am not taking sides here, but just for fairness, I think it should be
> noted that AWS EMR does exactly the same thing.
> We choose the EMR version (e.g., 6.4.0) and it has an associated Spark
> version (e.g., 3.1.2).
> The Spark version here is not the original Apache version but AWS Spark
> distribution.
>
> On Wed, Jun 7, 2023 at 8:24 PM Dongjoon Hyun 
> wrote:
>
>> I disagree with you in several ways.
>>
>> The following is not a *minor* change like the given examples
>> (alterations to the start-up and shutdown scripts, configuration files,
>> file layout etc.).
>>
>> > The change you cite meets the 4th point, minor change, made for
>> integration reasons.
>>
>> The following is also wrong. There is no such point of state of Apache
>> Spark 3.4.0 after 3.4.0 tag creation. Apache Spark community didn't allow
>> Scala reverting patches in both `master` branch and `branch-3.4`.
>>
>> > There is no known technical objection; this was after all at one point
>> the state of Apache Spark.
>>
>> Is the following your main point? So, you are selling a box "including
>> Harry Potter by J. K. Rolling whose main character is Barry instead of
>> Harry", but it's okay because you didn't sell the book itself? And, as a
>> cloud-vendor, you borrowed the box instead of selling it like private
>> libraries?
>>
>> > There is no standalone distribution of Apache Spark anywhere here.
>>
>> We are not asking a big thing. Why are you so reluctant to say you are
>> not "Apache Spark 3.4.0" by simply saying "Apache Spark 3.4.0-databricks".
>> What is the marketing reason here?
>>
>> Dongjoon.
>>
>>
>> On Wed, Jun 7, 2023 at 9:27 AM Sean Owen  wrote:
>>
>>> Hi Dongjoon, I think this conversation is not advancing anymore. I
>>> personally consider the matter closed unless you can find other support or
>>> respond with more specifics. While this perhaps should be on private@,
>>> I think it's not wrong as an instructive discussion on dev@.
>>>
>>> I don't believe you've made a clear argument about the problem, or how
>>> it relates specifically to policy. Nevertheless I will show you my logic.
>>>
>>> You are asserting that a vendor cannot call a product Apache Spark 3.4.0
>>> if it omits a patch updating a Scala maintenance version. This difference
>>> has no known impact on usage, as far as I can tell.
>>>
>>> Let's see what policy requires:
>>>
>>> 1/ All source code changes must meet at least one of the acceptable
>>> changes criteria set out below:
>>> - The change has accepted by the relevant Apache project community for
>>> inclusion in a future release. Note that the process used to accept changes
>>> and how that acceptance is documented varies between projects.
>>> - A change is a fix for an undisclosed security issue; and the fix is
>>> not publicly disclosed as as security fix; and the Apache project has been
>>> notified of the both issue and the proposed fix; and the PMC has rejected
>>> neither the vulnerability report nor the proposed fix.
>>> - A change is a fix for a bug; and the Apache project has been notified
>>> of both the bug and the proposed fix; and the PMC has rejected neither the
>>> bug report nor the proposed fix.
>>> - Minor changes (e.g. alterations to the start-up and shutdown scripts,
>>> configuration files, file layout etc.) to integrate with the target
>>> platform providing the Apache project has not objected to those changes.
>>>
>>> The change you cite meets the 4th point, minor change, made for
>>> integration reasons. There is no known technical objection; this was after
>>> all at one point the state of Apache Spark.
>>>
>>>
>>> 2/ A version number must be used that both clearly differentiates it
>>> from an Apache Software Foundation release and clearly identifies the
>>> Apache Software Foundation version on which the software is based.
>>>
>>> Keep in mind the product here is not "Apache Spark", but the "Databricks
>>> Runtime 13.1 (including Apache Spark 3.4.0)". That is, there is far more
>>> than a version number differentiating this product from Apache Spark. There
>>> is no standalone distribution of Apache Spark anywhere here. I believe that
>>> easily matches the intent.
>>>
>>>
>>> 3/ The documentation must clearly identify the Apache Software
>>> Foundation version on which the software is based.
>>>
>>> Clearly, yes.
>>>
>>>
>>> 4/ The end user expects that the distribution channel will back-port
>>> fixes. It is not necessary to back-port all fixes. Selection of fixes to
>>> back-port must be consistent with the update policy of that distribution
>>> channel.
>>>
>>> I think this is safe to say too. Indeed this explicitly contemplates not
>>> back-porting a change.
>>>
>>>
>>> Backing up, you can see from this document that the spirit of it is:
>>> don't include changes in your own Apache Foo x.y that aren't wanted by the
>>> project, and still 

Re: ASF policy violation and Scala version issues

[Grisha Weintraub]

Hi,

I am not taking sides here, but just for fairness, I think it should be
noted that AWS EMR does exactly the same thing.
We choose the EMR version (e.g., 6.4.0) and it has an associated Spark
version (e.g., 3.1.2).
The Spark version here is not the original Apache version but AWS Spark
distribution.

On Wed, Jun 7, 2023 at 8:24 PM Dongjoon Hyun 
wrote:

> I disagree with you in several ways.
>
> The following is not a *minor* change like the given examples (alterations
> to the start-up and shutdown scripts, configuration files, file layout
> etc.).
>
> > The change you cite meets the 4th point, minor change, made for
> integration reasons.
>
> The following is also wrong. There is no such point of state of Apache
> Spark 3.4.0 after 3.4.0 tag creation. Apache Spark community didn't allow
> Scala reverting patches in both `master` branch and `branch-3.4`.
>
> > There is no known technical objection; this was after all at one point
> the state of Apache Spark.
>
> Is the following your main point? So, you are selling a box "including
> Harry Potter by J. K. Rolling whose main character is Barry instead of
> Harry", but it's okay because you didn't sell the book itself? And, as a
> cloud-vendor, you borrowed the box instead of selling it like private
> libraries?
>
> > There is no standalone distribution of Apache Spark anywhere here.
>
> We are not asking a big thing. Why are you so reluctant to say you are not
> "Apache Spark 3.4.0" by simply saying "Apache Spark 3.4.0-databricks". What
> is the marketing reason here?
>
> Dongjoon.
>
>
> On Wed, Jun 7, 2023 at 9:27 AM Sean Owen  wrote:
>
>> Hi Dongjoon, I think this conversation is not advancing anymore. I
>> personally consider the matter closed unless you can find other support or
>> respond with more specifics. While this perhaps should be on private@, I
>> think it's not wrong as an instructive discussion on dev@.
>>
>> I don't believe you've made a clear argument about the problem, or how it
>> relates specifically to policy. Nevertheless I will show you my logic.
>>
>> You are asserting that a vendor cannot call a product Apache Spark 3.4.0
>> if it omits a patch updating a Scala maintenance version. This difference
>> has no known impact on usage, as far as I can tell.
>>
>> Let's see what policy requires:
>>
>> 1/ All source code changes must meet at least one of the acceptable
>> changes criteria set out below:
>> - The change has accepted by the relevant Apache project community for
>> inclusion in a future release. Note that the process used to accept changes
>> and how that acceptance is documented varies between projects.
>> - A change is a fix for an undisclosed security issue; and the fix is not
>> publicly disclosed as as security fix; and the Apache project has been
>> notified of the both issue and the proposed fix; and the PMC has rejected
>> neither the vulnerability report nor the proposed fix.
>> - A change is a fix for a bug; and the Apache project has been notified
>> of both the bug and the proposed fix; and the PMC has rejected neither the
>> bug report nor the proposed fix.
>> - Minor changes (e.g. alterations to the start-up and shutdown scripts,
>> configuration files, file layout etc.) to integrate with the target
>> platform providing the Apache project has not objected to those changes.
>>
>> The change you cite meets the 4th point, minor change, made for
>> integration reasons. There is no known technical objection; this was after
>> all at one point the state of Apache Spark.
>>
>>
>> 2/ A version number must be used that both clearly differentiates it from
>> an Apache Software Foundation release and clearly identifies the Apache
>> Software Foundation version on which the software is based.
>>
>> Keep in mind the product here is not "Apache Spark", but the "Databricks
>> Runtime 13.1 (including Apache Spark 3.4.0)". That is, there is far more
>> than a version number differentiating this product from Apache Spark. There
>> is no standalone distribution of Apache Spark anywhere here. I believe that
>> easily matches the intent.
>>
>>
>> 3/ The documentation must clearly identify the Apache Software Foundation
>> version on which the software is based.
>>
>> Clearly, yes.
>>
>>
>> 4/ The end user expects that the distribution channel will back-port
>> fixes. It is not necessary to back-port all fixes. Selection of fixes to
>> back-port must be consistent with the update policy of that distribution
>> channel.
>>
>> I think this is safe to say too. Indeed this explicitly contemplates not
>> back-porting a change.
>>
>>
>> Backing up, you can see from this document that the spirit of it is:
>> don't include changes in your own Apache Foo x.y that aren't wanted by the
>> project, and still call it Apache Foo x.y. I don't believe your case
>> matches this spirit either.
>>
>> I do think it's not crazy to suggest, hey vendor, would you call this
>> "Apache Spark + patches" or ".vendor123". But that's at best a suggestion,

Re: ASF policy violation and Scala version issues

[Mich Talebzadeh]

OK, is this the crux of the matter?

We are not asking a big thing ...

First, who are we here? members?

In my opinion, without being overly specific, this discussion has lost its
objectivity. However, with reference to your point, I am sure, a simple
vote will clarify the position in a fairer way.

for me + 1 for Sean's assertions, meaning I agree with Sean to move on and
close this discussion.

Respectfully

Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Wed, 7 Jun 2023 at 18:24, Dongjoon Hyun  wrote:

> I disagree with you in several ways.
>
> The following is not a *minor* change like the given examples (alterations
> to the start-up and shutdown scripts, configuration files, file layout
> etc.).
>
> > The change you cite meets the 4th point, minor change, made for
> integration reasons.
>
> The following is also wrong. There is no such point of state of Apache
> Spark 3.4.0 after 3.4.0 tag creation. Apache Spark community didn't allow
> Scala reverting patches in both `master` branch and `branch-3.4`.
>
> > There is no known technical objection; this was after all at one point
> the state of Apache Spark.
>
> Is the following your main point? So, you are selling a box "including
> Harry Potter by J. K. Rolling whose main character is Barry instead of
> Harry", but it's okay because you didn't sell the book itself? And, as a
> cloud-vendor, you borrowed the box instead of selling it like private
> libraries?
>
> > There is no standalone distribution of Apache Spark anywhere here.
>
> We are not asking a big thing. Why are you so reluctant to say you are not
> "Apache Spark 3.4.0" by simply saying "Apache Spark 3.4.0-databricks". What
> is the marketing reason here?
>
> Dongjoon.
>
>
> On Wed, Jun 7, 2023 at 9:27 AM Sean Owen  wrote:
>
>> Hi Dongjoon, I think this conversation is not advancing anymore. I
>> personally consider the matter closed unless you can find other support or
>> respond with more specifics. While this perhaps should be on private@, I
>> think it's not wrong as an instructive discussion on dev@.
>>
>> I don't believe you've made a clear argument about the problem, or how it
>> relates specifically to policy. Nevertheless I will show you my logic.
>>
>> You are asserting that a vendor cannot call a product Apache Spark 3.4.0
>> if it omits a patch updating a Scala maintenance version. This difference
>> has no known impact on usage, as far as I can tell.
>>
>> Let's see what policy requires:
>>
>> 1/ All source code changes must meet at least one of the acceptable
>> changes criteria set out below:
>> - The change has accepted by the relevant Apache project community for
>> inclusion in a future release. Note that the process used to accept changes
>> and how that acceptance is documented varies between projects.
>> - A change is a fix for an undisclosed security issue; and the fix is not
>> publicly disclosed as as security fix; and the Apache project has been
>> notified of the both issue and the proposed fix; and the PMC has rejected
>> neither the vulnerability report nor the proposed fix.
>> - A change is a fix for a bug; and the Apache project has been notified
>> of both the bug and the proposed fix; and the PMC has rejected neither the
>> bug report nor the proposed fix.
>> - Minor changes (e.g. alterations to the start-up and shutdown scripts,
>> configuration files, file layout etc.) to integrate with the target
>> platform providing the Apache project has not objected to those changes.
>>
>> The change you cite meets the 4th point, minor change, made for
>> integration reasons. There is no known technical objection; this was after
>> all at one point the state of Apache Spark.
>>
>>
>> 2/ A version number must be used that both clearly differentiates it from
>> an Apache Software Foundation release and clearly identifies the Apache
>> Software Foundation version on which the software is based.
>>
>> Keep in mind the product here is not "Apache Spark", but the "Databricks
>> Runtime 13.1 (including Apache Spark 3.4.0)". That is, there is far more
>> than a version number differentiating this product from Apache Spark. There
>> is no standalone distribution of Apache Spark anywhere here. I believe that
>> easily matches the intent.
>>
>>
>> 3/ The documentation must clearly identify the Apache Software Foundation
>> version on which the software is based.
>>
>> Clearly, yes.
>>
>>
>> 4/ The end user expects that the distribution channel will 

Re: JDK version support policy?

[Sean Owen]

I also generally perceive that, after Java 9, there is much less breaking
change. So working on Java 11 probably means it works on 20, or can be
easily made to without pain. Like I think the tweaks for Java 17 were quite
small.

Targeting Java >11 excludes Java 11 users and probably wouldn't buy much.
Keeping the support probably doesn't interfere with working on much newer
JVMs either.

On Wed, Jun 7, 2023, 12:29 PM Holden Karau  wrote:

> So JDK 11 is still supported in open JDK until 2026, I'm not sure if we're
> going to see enough folks moving to JRE17 by the Spark 4 release unless we
> have a strong benefit from dropping 11 support I'd be inclined to keep it.
>
> On Tue, Jun 6, 2023 at 9:08 PM Dongjoon Hyun  wrote:
>
>> I'm also +1 on dropping both Java 8 and 11 in Apache Spark 4.0, too.
>>
>> Dongjoon.
>>
>> On 2023/06/07 02:42:19 yangjie01 wrote:
>> > +1 on dropping Java 8 in Spark 4.0, and I even hope Spark 4.0 can only
>> support Java 17 and the upcoming Java 21.
>> >
>> > 发件人: Denny Lee 
>> > 日期: 2023年6月7日 星期三 07:10
>> > 收件人: Sean Owen 
>> > 抄送: David Li , "dev@spark.apache.org" <
>> dev@spark.apache.org>
>> > 主题: Re: JDK version support policy?
>> >
>> > +1 on dropping Java 8 in Spark 4.0, saying this as a fan of the
>> fast-paced (positive) updates to Arrow, eh?!
>> >
>> > On Tue, Jun 6, 2023 at 4:02 PM Sean Owen > sro...@gmail.com>> wrote:
>> > I haven't followed this discussion closely, but I think we could/should
>> drop Java 8 in Spark 4.0, which is up next after 3.5?
>> >
>> > On Tue, Jun 6, 2023 at 2:44 PM David Li > lidav...@apache.org>> wrote:
>> > Hello Spark developers,
>> >
>> > I'm from the Apache Arrow project. We've discussed Java version support
>> [1], and crucially, whether to continue supporting Java 8 or not. As Spark
>> is a big user of Arrow in Java, I was curious what Spark's policy here was.
>> >
>> > If Spark intends to stay on Java 8, for instance, we may also want to
>> stay on Java 8 or otherwise provide some supported version of Arrow for
>> Java 8.
>> >
>> > We've seen dependencies dropping or planning to drop support. gRPC may
>> drop Java 8 at any time [2], possibly this September [3], which may affect
>> Spark (due to Spark Connect). And today we saw that Arrow had issues
>> running tests with Mockito on Java 20, but we couldn't update Mockito since
>> it had dropped Java 8 support. (We pinned the JDK version in that CI
>> pipeline for now.)
>> >
>> > So at least, I am curious if Arrow could start the long process of
>> migrating Java versions without impacting Spark, or if we should continue
>> to cooperate. Arrow Java doesn't see quite so much activity these days, so
>> it's not quite critical, but it's possible that these dependency issues
>> will start to affect us more soon. And looking forward, Java is working on
>> APIs that should also allow us to ditch the --add-opens flag requirement
>> too.
>> >
>> > [1]: https://lists.apache.org/thread/phpgpydtt3yrgnncdyv4qdq1gf02s0yj<
>> https://mailshield.baidu.com/check?q=Nz%2bGj2hdKguk92URjA7sg0PfbSN%2fXUIMgrHTmW45gOOKEr3Shre45B7TRzhEpb%2baVsnyuRL%2fl%2f0cu7IVGHunSGDVnxM%3d
>> >
>> > [2]:
>> https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md<
>> https://mailshield.baidu.com/check?q=s89S3eo8GCJkV7Mpx7aG1SXId7uCRYGjQMA6DeLuX9duS86LhIODZMJfeFdGMWdFzJ8S7minyHoC7mCrzHagbJXCXYTBH%2fpZBpfTbw%3d%3d
>> >
>> > [3]: https://github.com/grpc/grpc-java/issues/9386<
>> https://mailshield.baidu.com/check?q=R0HtWZIkY5eIxpz8jtqHLzd0ugNbcaXIKW2LbUUxpIn0t9Y9yAhuHPuZ4buryfNwRnnJTA%3d%3d
>> >
>> >
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: JDK version support policy?

[Holden Karau]

So JDK 11 is still supported in open JDK until 2026, I'm not sure if we're
going to see enough folks moving to JRE17 by the Spark 4 release unless we
have a strong benefit from dropping 11 support I'd be inclined to keep it.

On Tue, Jun 6, 2023 at 9:08 PM Dongjoon Hyun  wrote:

> I'm also +1 on dropping both Java 8 and 11 in Apache Spark 4.0, too.
>
> Dongjoon.
>
> On 2023/06/07 02:42:19 yangjie01 wrote:
> > +1 on dropping Java 8 in Spark 4.0, and I even hope Spark 4.0 can only
> support Java 17 and the upcoming Java 21.
> >
> > 发件人: Denny Lee 
> > 日期: 2023年6月7日 星期三 07:10
> > 收件人: Sean Owen 
> > 抄送: David Li , "dev@spark.apache.org" <
> dev@spark.apache.org>
> > 主题: Re: JDK version support policy?
> >
> > +1 on dropping Java 8 in Spark 4.0, saying this as a fan of the
> fast-paced (positive) updates to Arrow, eh?!
> >
> > On Tue, Jun 6, 2023 at 4:02 PM Sean Owen  sro...@gmail.com>> wrote:
> > I haven't followed this discussion closely, but I think we could/should
> drop Java 8 in Spark 4.0, which is up next after 3.5?
> >
> > On Tue, Jun 6, 2023 at 2:44 PM David Li  lidav...@apache.org>> wrote:
> > Hello Spark developers,
> >
> > I'm from the Apache Arrow project. We've discussed Java version support
> [1], and crucially, whether to continue supporting Java 8 or not. As Spark
> is a big user of Arrow in Java, I was curious what Spark's policy here was.
> >
> > If Spark intends to stay on Java 8, for instance, we may also want to
> stay on Java 8 or otherwise provide some supported version of Arrow for
> Java 8.
> >
> > We've seen dependencies dropping or planning to drop support. gRPC may
> drop Java 8 at any time [2], possibly this September [3], which may affect
> Spark (due to Spark Connect). And today we saw that Arrow had issues
> running tests with Mockito on Java 20, but we couldn't update Mockito since
> it had dropped Java 8 support. (We pinned the JDK version in that CI
> pipeline for now.)
> >
> > So at least, I am curious if Arrow could start the long process of
> migrating Java versions without impacting Spark, or if we should continue
> to cooperate. Arrow Java doesn't see quite so much activity these days, so
> it's not quite critical, but it's possible that these dependency issues
> will start to affect us more soon. And looking forward, Java is working on
> APIs that should also allow us to ditch the --add-opens flag requirement
> too.
> >
> > [1]: https://lists.apache.org/thread/phpgpydtt3yrgnncdyv4qdq1gf02s0yj<
> https://mailshield.baidu.com/check?q=Nz%2bGj2hdKguk92URjA7sg0PfbSN%2fXUIMgrHTmW45gOOKEr3Shre45B7TRzhEpb%2baVsnyuRL%2fl%2f0cu7IVGHunSGDVnxM%3d
> >
> > [2]:
> https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md<
> https://mailshield.baidu.com/check?q=s89S3eo8GCJkV7Mpx7aG1SXId7uCRYGjQMA6DeLuX9duS86LhIODZMJfeFdGMWdFzJ8S7minyHoC7mCrzHagbJXCXYTBH%2fpZBpfTbw%3d%3d
> >
> > [3]: https://github.com/grpc/grpc-java/issues/9386<
> https://mailshield.baidu.com/check?q=R0HtWZIkY5eIxpz8jtqHLzd0ugNbcaXIKW2LbUUxpIn0t9Y9yAhuHPuZ4buryfNwRnnJTA%3d%3d
> >
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: ASF policy violation and Scala version issues

[Dongjoon Hyun]

I disagree with you in several ways.

The following is not a *minor* change like the given examples (alterations
to the start-up and shutdown scripts, configuration files, file layout
etc.).

> The change you cite meets the 4th point, minor change, made for
integration reasons.

The following is also wrong. There is no such point of state of Apache
Spark 3.4.0 after 3.4.0 tag creation. Apache Spark community didn't allow
Scala reverting patches in both `master` branch and `branch-3.4`.

> There is no known technical objection; this was after all at one point
the state of Apache Spark.

Is the following your main point? So, you are selling a box "including
Harry Potter by J. K. Rolling whose main character is Barry instead of
Harry", but it's okay because you didn't sell the book itself? And, as a
cloud-vendor, you borrowed the box instead of selling it like private
libraries?

> There is no standalone distribution of Apache Spark anywhere here.

We are not asking a big thing. Why are you so reluctant to say you are not
"Apache Spark 3.4.0" by simply saying "Apache Spark 3.4.0-databricks". What
is the marketing reason here?

Dongjoon.


On Wed, Jun 7, 2023 at 9:27 AM Sean Owen  wrote:

> Hi Dongjoon, I think this conversation is not advancing anymore. I
> personally consider the matter closed unless you can find other support or
> respond with more specifics. While this perhaps should be on private@, I
> think it's not wrong as an instructive discussion on dev@.
>
> I don't believe you've made a clear argument about the problem, or how it
> relates specifically to policy. Nevertheless I will show you my logic.
>
> You are asserting that a vendor cannot call a product Apache Spark 3.4.0
> if it omits a patch updating a Scala maintenance version. This difference
> has no known impact on usage, as far as I can tell.
>
> Let's see what policy requires:
>
> 1/ All source code changes must meet at least one of the acceptable
> changes criteria set out below:
> - The change has accepted by the relevant Apache project community for
> inclusion in a future release. Note that the process used to accept changes
> and how that acceptance is documented varies between projects.
> - A change is a fix for an undisclosed security issue; and the fix is not
> publicly disclosed as as security fix; and the Apache project has been
> notified of the both issue and the proposed fix; and the PMC has rejected
> neither the vulnerability report nor the proposed fix.
> - A change is a fix for a bug; and the Apache project has been notified of
> both the bug and the proposed fix; and the PMC has rejected neither the bug
> report nor the proposed fix.
> - Minor changes (e.g. alterations to the start-up and shutdown scripts,
> configuration files, file layout etc.) to integrate with the target
> platform providing the Apache project has not objected to those changes.
>
> The change you cite meets the 4th point, minor change, made for
> integration reasons. There is no known technical objection; this was after
> all at one point the state of Apache Spark.
>
>
> 2/ A version number must be used that both clearly differentiates it from
> an Apache Software Foundation release and clearly identifies the Apache
> Software Foundation version on which the software is based.
>
> Keep in mind the product here is not "Apache Spark", but the "Databricks
> Runtime 13.1 (including Apache Spark 3.4.0)". That is, there is far more
> than a version number differentiating this product from Apache Spark. There
> is no standalone distribution of Apache Spark anywhere here. I believe that
> easily matches the intent.
>
>
> 3/ The documentation must clearly identify the Apache Software Foundation
> version on which the software is based.
>
> Clearly, yes.
>
>
> 4/ The end user expects that the distribution channel will back-port
> fixes. It is not necessary to back-port all fixes. Selection of fixes to
> back-port must be consistent with the update policy of that distribution
> channel.
>
> I think this is safe to say too. Indeed this explicitly contemplates not
> back-porting a change.
>
>
> Backing up, you can see from this document that the spirit of it is: don't
> include changes in your own Apache Foo x.y that aren't wanted by the
> project, and still call it Apache Foo x.y. I don't believe your case
> matches this spirit either.
>
> I do think it's not crazy to suggest, hey vendor, would you call this
> "Apache Spark + patches" or ".vendor123". But that's at best a suggestion,
> and I think it does nothing in particular for users. You've made the
> suggestion, and I do not see some police action from the PMC must follow.
>
>
> I think you're simply objecting to a vendor choice, but that is not
> on-topic here unless you can specifically rebut the reasoning above and
> show it's connected.
>
>
> On Wed, Jun 7, 2023 at 11:02 AM Dongjoon Hyun  wrote:
>
>> Sean, it seems that you are confused here. We are not talking about your
>> upper system (the 

Re: ASF policy violation and Scala version issues

[Sean Owen]

Hi Dongjoon, I think this conversation is not advancing anymore. I
personally consider the matter closed unless you can find other support or
respond with more specifics. While this perhaps should be on private@, I
think it's not wrong as an instructive discussion on dev@.

I don't believe you've made a clear argument about the problem, or how it
relates specifically to policy. Nevertheless I will show you my logic.

You are asserting that a vendor cannot call a product Apache Spark 3.4.0 if
it omits a patch updating a Scala maintenance version. This difference has
no known impact on usage, as far as I can tell.

Let's see what policy requires:

1/ All source code changes must meet at least one of the acceptable changes
criteria set out below:
- The change has accepted by the relevant Apache project community for
inclusion in a future release. Note that the process used to accept changes
and how that acceptance is documented varies between projects.
- A change is a fix for an undisclosed security issue; and the fix is not
publicly disclosed as as security fix; and the Apache project has been
notified of the both issue and the proposed fix; and the PMC has rejected
neither the vulnerability report nor the proposed fix.
- A change is a fix for a bug; and the Apache project has been notified of
both the bug and the proposed fix; and the PMC has rejected neither the bug
report nor the proposed fix.
- Minor changes (e.g. alterations to the start-up and shutdown scripts,
configuration files, file layout etc.) to integrate with the target
platform providing the Apache project has not objected to those changes.

The change you cite meets the 4th point, minor change, made for integration
reasons. There is no known technical objection; this was after all at one
point the state of Apache Spark.


2/ A version number must be used that both clearly differentiates it from
an Apache Software Foundation release and clearly identifies the Apache
Software Foundation version on which the software is based.

Keep in mind the product here is not "Apache Spark", but the "Databricks
Runtime 13.1 (including Apache Spark 3.4.0)". That is, there is far more
than a version number differentiating this product from Apache Spark. There
is no standalone distribution of Apache Spark anywhere here. I believe that
easily matches the intent.


3/ The documentation must clearly identify the Apache Software Foundation
version on which the software is based.

Clearly, yes.


4/ The end user expects that the distribution channel will back-port fixes.
It is not necessary to back-port all fixes. Selection of fixes to back-port
must be consistent with the update policy of that distribution channel.

I think this is safe to say too. Indeed this explicitly contemplates not
back-porting a change.


Backing up, you can see from this document that the spirit of it is: don't
include changes in your own Apache Foo x.y that aren't wanted by the
project, and still call it Apache Foo x.y. I don't believe your case
matches this spirit either.

I do think it's not crazy to suggest, hey vendor, would you call this
"Apache Spark + patches" or ".vendor123". But that's at best a suggestion,
and I think it does nothing in particular for users. You've made the
suggestion, and I do not see some police action from the PMC must follow.


I think you're simply objecting to a vendor choice, but that is not
on-topic here unless you can specifically rebut the reasoning above and
show it's connected.


On Wed, Jun 7, 2023 at 11:02 AM Dongjoon Hyun  wrote:

> Sean, it seems that you are confused here. We are not talking about your
> upper system (the notebook environment). We are talking about the
> submodule, "Apache Spark 3.4.0-databricks". Whatever you call it, both of
> us knows "Apache Spark 3.4.0-databricks" is different from "Apache Spark
> 3.4.0". You should not use "3.4.0" at your subsystem.
>
> > This also is aimed at distributions of "Apache Foo", not products that
> > "include Apache Foo", which are clearly not Apache Foo.
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: ASF policy violation and Scala version issues

[Dongjoon Hyun]

Sean, it seems that you are confused here. We are not talking about your upper 
system (the notebook environment). We are talking about the submodule, "Apache 
Spark 3.4.0-databricks". Whatever you call it, both of us knows "Apache Spark 
3.4.0-databricks" is different from "Apache Spark 3.4.0". You should not use 
"3.4.0" at your subsystem.

> This also is aimed at distributions of "Apache Foo", not products that
> "include Apache Foo", which are clearly not Apache Foo.

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: ASF policy violation and Scala version issues

[Sean Owen]

(With consent, shall we move this to the PMC list?)

No, I don't think that's what this policy says.

First, could you please be more specific here? why do you think a certain
release is at odds with this?
Because so far you've mentioned, I think, not taking a Scala maintenance
release update.

But this says things like:

The source code on which the software is based must either be identical to
an Apache Software Foundation source code release or all of the following
must also be true:
  ...
  - The end user expects that the distribution channel will back-port
fixes. It is not necessary to back-port all fixes. Selection of fixes to
back-port must be consistent with the update policy of that distribution
channel.

That describes what you're talking about.

This also is aimed at distributions of "Apache Foo", not products that
"include Apache Foo", which are clearly not Apache Foo.
The spirit of it is, more generally: don't keep new features and fixes to
yourself. That does not seem to apply here.

On Tue, Jun 6, 2023 at 11:34 PM Dongjoon Hyun 
wrote:

> Hi, All and Matei (as the Chair of Spark PMC).
>
> For the ASF policy violation part, here is a legal recommendation
> documentation (draft) from `legal-discuss@`.
>
> https://www.apache.org/foundation/marks/downstream.html#source
>
> > A version number must be used that both clearly differentiates it from
> an Apache Software Foundation release and clearly identifies the Apache
> Software Foundation version on which the software is based.
>
> In short, Databricks should not claim its product like "Apache Spark
> 3.4.0". The version number should clearly differentiate it from Apache
> Spark 3.4.0. I hope we can conclude this together in this way and move our
> focus forward to the other remaining issues.
>
> To Matei, could you do the legal follow-up officially with Databricks with
> the above info?
>
> If there is a person to do this, I believe you are the best person to
> drive this.
>
> Thank you in advance.
>
> Dongjoon.
>
>
> On Tue, Jun 6, 2023 at 2:49 PM Dongjoon Hyun  wrote:
>
>> It goes to "legal-discuss@".
>>
>> https://lists.apache.org/thread/mzhggd0rpz8t4d7vdsbhkp38mvd3lty4
>>
>> I hope we can conclude the legal part clearly and shortly in one way or
>> another which we will follow with confidence.
>>
>> Dongjoon
>>
>> On 2023/06/06 20:06:42 Dongjoon Hyun wrote:
>> > Thank you, Sean, Mich, Holden, again.
>> >
>> > For this specific part, let's ask the ASF board via bo...@apache.org to
>> > find a right answer because it's a controversial legal issue here.
>> >
>> > > I think you'd just prefer Databricks make a different choice, which is
>> > legitimate, but, an issue to take up with Databricks, not here.
>> >
>> > Dongjoon.
>> >
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>