[
https://issues.apache.org/jira/browse/SPARK-32530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17188816#comment-17188816
]
Dennis Jaheruddin commented on SPARK-32530:
-------------------------------------------
Please note that a significant number of on prem users is running spark on HDP
2 and HDP 3, they will be on 2.3.2. CDH 6 is on 2.4.0. (I won't even mention
what version CDH5 users are on). In both cases there is a successor announced,
but most hadoop users will be on lower versions than you plan to support for
the next 12-18 months.
This may help if you want to tap into a larger audience.
> SPIP: Kotlin support for Apache Spark
> -------------------------------------
>
> Key: SPARK-32530
> URL: https://issues.apache.org/jira/browse/SPARK-32530
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.1
> Reporter: Pasha Finkeshteyn
> Priority: Major
>
> h2. Background and motivation
> Kotlin is a cross-platform, statically typed, general-purpose JVM language.
> In the last year more than 5 million developers have used Kotlin in mobile,
> backend, frontend and scientific development. The number of Kotlin developers
> grows rapidly every year.
> * [According to
> redmonk|https://redmonk.com/sogrady/2020/02/28/language-rankings-1-20/]:
> "Kotlin, the second fastest growing language we’ve seen outside of Swift,
> made a big splash a year ago at this time when it vaulted eight full spots up
> the list."
> * [According to snyk.io|https://snyk.io/wp-content/uploads/jvm_2020.pdf],
> Kotlin is the second most popular language on the JVM
> * [According to
> StackOverflow|https://insights.stackoverflow.com/survey/2020] Kotlin’s share
> increased by 7.8% in 2020.
> We notice the increasing usage of Kotlin in data analysis ([6% of users in
> 2020|https://www.jetbrains.com/lp/devecosystem-2020/kotlin/], as opposed to
> 2% in 2019) and machine learning (3% of users in 2020, as opposed to 0% in
> 2019), and we expect these numbers to continue to grow.
> We, authors of this SPIP, strongly believe that making Kotlin API officially
> available to developers can bring new users to Apache Spark and help some of
> the existing users.
> h2. Goals
> The goal of this project is to bring first-class support for Kotlin language
> into the Apache Spark project. We’re going to achieve this by adding one more
> module to the current Apache Spark distribution.
> h2. Non-goals
> There is no goal to replace any existing language support or to change any
> existing Apache Spark API.
> At this time, there is no goal to support non-core APIs of Apache Spark like
> Spark ML and Spark structured streaming. This may change in the future based
> on community feedback.
> There is no goal to provide CLI for Kotlin for Apache Spark, this will be a
> separate SPIP.
> There is no goal to provide support for Apache Spark < 3.0.0.
> h2. Current implementation
> A working prototype is available at
> [https://github.com/JetBrains/kotlin-spark-api]. It has been tested inside
> JetBrains and by early adopters.
> h2. What are the risks?
> There is always a risk that this product won’t get enough popularity and will
> bring more costs than benefits. It can be mitigated by the fact that we don't
> need to change any existing API and support can be potentially dropped at any
> time.
> We also believe that existing API is rather low maintenance. It does not
> bring anything more complex than already exists in the Spark codebase.
> Furthermore, the implementation is compact - less than 2000 lines of code.
> We are committed to maintaining, improving and evolving the API based on
> feedback from both Spark and Kotlin communities. As the Kotlin data community
> continues to grow, we see Kotlin API for Apache Spark as an important part in
> the evolving Kotlin ecosystem, and intend to fully support it.
> h2. How long will it take?
> A working implementation is already available, and if the community will
> have any proposal of changes for this implementation to be improved, these
> can be implemented quickly — in weeks if not days.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
