I want to add two updates on this thread.
First, Ammonite library issue is resolved for Scala 2.12/2.13. For Scala 3, we
can talk later in the scope of Spark 4.
SPARK-44041 Upgrade Ammonite to 2.5.9
This unblocked the following and we start to evaluate them.
SPARK-43832 Upgrade Scala
Let me add my answers about a few Scala questions, Jungtaek.
> Are we concerned that a library does not release a new version
> which bumps the Scala version, which the Scala version is
> announced in less than a week?
No, we have concerns about the newly introduced disability
in the Apache
Yes, you're right.
发件人: Jungtaek Lim
日期: 2023年6月12日 星期一 11:37
收件人: Dongjoon Hyun
抄送: yangjie01 , Grisha Weintraub
, Nan Zhu , Sean Owen
, "dev@spark.apache.org"
主题: Re: ASF policy violation and Scala version issues
Are we concerned that a library does not release a new version w
it with a Scala-shell based implementation
>> - Move `connector/connect/client/jvm/pom.xml` outside from Spark repo.
>> Maybe, we can put it into the new repo like Rust and Go client.
>>
>> ```
>>
>> *发件人**: *Grisha Weintraub
>> *日期**: *2023年6月8日 星期四 04:05
gt;
> ```
>
> *发件人**: *Grisha Weintraub
> *日期**: *2023年6月8日 星期四 04:05
> *收件人**: *Dongjoon Hyun
> *抄送**: *Nan Zhu , Sean Owen , "
> dev@spark.apache.org"
> *主题**: *Re: ASF policy violation and Scala version issues
>
>
>
> Dongjoon,
>
>
>
> I followe
t;dev@spark.apache.org"
主题: Re: ASF policy violation and Scala version issues
Dongjoon,
I followed the conversation, and in my opinion, your concern is totally legit.
It just feels that the discussion is focused solely on Databricks, and as I
said above, the same issue occurs in other vendor
Dongjoon,
I followed the conversation, and in my opinion, your concern is totally
legit.
It just feels that the discussion is focused solely on Databricks, and as I
said above, the same issue occurs in other vendors as well.
On Wed, Jun 7, 2023 at 10:28 PM Dongjoon Hyun
wrote:
> To Grisha, we
To Grisha, we are talking about what is the right way and how to comply
with ASF legal advice which I shared in this thread from "legal-discuss@"
mailing thread.
https://lists.apache.org/thread/mzhggd0rpz8t4d7vdsbhkp38mvd3lty4
(legal-discuss@)
Yes, in Spark UI you have it as "3.1.2-amazon", but when you create a
cluster it's just Spark 3.1.2.
On Wed, Jun 7, 2023 at 10:05 PM Nan Zhu wrote:
>
> for EMR, I think they show 3.1.2-amazon in Spark UI, no?
>
>
> On Wed, Jun 7, 2023 at 11:30 Grisha Weintraub
> wrote:
>
>> Hi,
>>
>> I am not
for EMR, I think they show 3.1.2-amazon in Spark UI, no?
On Wed, Jun 7, 2023 at 11:30 Grisha Weintraub
wrote:
> Hi,
>
> I am not taking sides here, but just for fairness, I think it should be
> noted that AWS EMR does exactly the same thing.
> We choose the EMR version (e.g., 6.4.0) and it
Hi,
I am not taking sides here, but just for fairness, I think it should be
noted that AWS EMR does exactly the same thing.
We choose the EMR version (e.g., 6.4.0) and it has an associated Spark
version (e.g., 3.1.2).
The Spark version here is not the original Apache version but AWS Spark
OK, is this the crux of the matter?
We are not asking a big thing ...
First, who are we here? members?
In my opinion, without being overly specific, this discussion has lost its
objectivity. However, with reference to your point, I am sure, a simple
vote will clarify the position in a fairer
I disagree with you in several ways.
The following is not a *minor* change like the given examples (alterations
to the start-up and shutdown scripts, configuration files, file layout
etc.).
> The change you cite meets the 4th point, minor change, made for
integration reasons.
The following is
Hi Dongjoon, I think this conversation is not advancing anymore. I
personally consider the matter closed unless you can find other support or
respond with more specifics. While this perhaps should be on private@, I
think it's not wrong as an instructive discussion on dev@.
I don't believe you've
Sean, it seems that you are confused here. We are not talking about your upper
system (the notebook environment). We are talking about the submodule, "Apache
Spark 3.4.0-databricks". Whatever you call it, both of us knows "Apache Spark
3.4.0-databricks" is different from "Apache Spark 3.4.0".
(With consent, shall we move this to the PMC list?)
No, I don't think that's what this policy says.
First, could you please be more specific here? why do you think a certain
release is at odds with this?
Because so far you've mentioned, I think, not taking a Scala maintenance
release update.
Hi, All and Matei (as the Chair of Spark PMC).
For the ASF policy violation part, here is a legal recommendation
documentation (draft) from `legal-discuss@`.
https://www.apache.org/foundation/marks/downstream.html#source
> A version number must be used that both clearly differentiates it from
Hello,
This explanation is splendidly detailed and requires further understanding.
However, on a first thought with regard to the point raised below and I
quote:
"... There is a company claiming something non-Apache like "Apache Spark
3.4.0 minus SPARK-40436" with the name "Apache Spark 3.4.0."
It goes to "legal-discuss@".
https://lists.apache.org/thread/mzhggd0rpz8t4d7vdsbhkp38mvd3lty4
I hope we can conclude the legal part clearly and shortly in one way or another
which we will follow with confidence.
Dongjoon
On 2023/06/06 20:06:42 Dongjoon Hyun wrote:
> Thank you, Sean, Mich,
Thank you, Sean, Mich, Holden, again.
For this specific part, let's ask the ASF board via bo...@apache.org to
find a right answer because it's a controversial legal issue here.
> I think you'd just prefer Databricks make a different choice, which is
legitimate, but, an issue to take up with
So I think if the Spark PMC wants to ask Databricks something that could be
reasonable (although I'm a little fuzzy as to the ask), but that
conversation might belong on private@ (I could be wrong of course).
On Tue, Jun 6, 2023 at 3:29 AM Mich Talebzadeh
wrote:
> I concur with you Sean.
>
> If
I concur with you Sean.
If I understand correctly the point raised by the thread owner, in
heterogeneous environments that we work, it is up to the practitioner to
ensure that there is version compatibility among OS versions, spark version
and the target artefact in consideration. For example if
I think the issue is whether a distribution of Spark is so materially
different from OSS that it causes problems for the larger community of
users. There's a legitimate question of whether such a thing can be called
"Apache Spark + changes", as describing it that way becomes meaningfully
Hi, Sean.
"+ patches" or "powered by Apache Spark 3.4.0" is not a problem as you
mentioned. For the record, I also didn't bring up any old story here.
> "Apache Spark 3.4.0 + patches"
However, "including Apache Spark 3.4.0" still causes confusion even in a
different way because of those missing
On Mon, Jun 5, 2023 at 12:01 PM Dongjoon Hyun
wrote:
> 1. For the naming, yes, but the company should use different version
> numbers instead of the exact "3.4.0". As I shared the screenshot in my
> previous email, the company exposes "Apache Spark 3.4.0" exactly because
> they build their
Thank you, Sean.
I'll reply as a comment for some areas first.
> I believe releasing "Apache Foo X.Y + patches" is acceptable,
> if it is substantially Apache Foo X.Y.
1. For the naming, yes, but the company should use different version
numbers instead of the exact "3.4.0". As I shared the
1/ Regarding naming - I believe releasing "Apache Foo X.Y + patches" is
acceptable, if it is substantially Apache Foo X.Y. This is common practice
for downstream vendors. It's fair nominative use. The principle here is
consumer confusion. Is anyone substantially misled? Here I don't think so.
I
Hi, All and Matei (as the Chair of Apache Spark PMC).
Sorry for a long email, I want to share two topics and corresponding action
items.
You can go to "Section 3: Action Items" directly for the conclusion.
### 1. ASF Policy Violation ###
ASF has a rule for "MAY I CALL MY MODIFIED CODE
28 matches
Mail list logo
