Re: Kotlin Spark API
Hi Stephen,
Thank you so much for finding time for looking at our examples! Yes, we've
tried to implement as clean design of API as possible and are constantly
looking for ways to make it even more readable, clear and friendly.
And as Maria already stated we welcome any feedback!
On 20/07/14 01:55PM, Stephen Boesch wrote:
> I just looked at the examples.
> https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples
> These look v nice! V concise yet flexible. I like the ability to do
> inline *side-effects. *E.g. caching or printing or showDs()
>
> package org.jetbrains.spark.api.examples
> import org.apache.spark.sql.Row
> import org.jetbrains.spark.api.*
>
> fun main() {
> withSpark {
> val sd = dsOf(1, 2, 3)
> sd.createOrReplaceTempView("ds")
> spark.sql("select * from ds")
> .withCached {
> println("asList: ${toList()}")
> println("asArray: ${toArray().contentToString()}")
> this
> }
> .to()
> .withCached {
> println("typed collect: " + (collect() as
> Array).contentToString())
> println("type collectAsList: " + collectAsList())
> }
>
> dsOf(1, 2, 3)
> .map { c(it, it + 1, it + 2) }
> .to()
> .select("_1")
> .collectAsList()
> .forEach { println(it) }
> }
> }
>
>
> So that shows some of the niceness of kotlin: intuitive type conversion
> `to`/`to` and `dsOf( list)`- and also the inlining of the side
> effects. Overall concise and pleasant to read.
>
>
> On Tue, 14 Jul 2020 at 12:18, Stephen Boesch wrote:
>
> > I started with scala/spark in 2012 and scala has been my go-to language
> > for six years. But I heartily applaud this direction. Kotlin is more like a
> > simplified Scala - with the benefits that brings - than a simplified java.
> > I particularly like the simplified / streamlined collections classes.
> >
> > Really looking forward to this development.
> >
> > On Tue, 14 Jul 2020 at 10:42, Maria Khalusova wrote:
> >
> >> Hi folks,
> >>
> >> We would love your feedback on the new Kotlin Spark API that we are
> >> working on: https://github.com/JetBrains/kotlin-spark-api.
> >>
> >> Why Kotlin Spark API? Kotlin developers can already use Kotlin with the
> >> existing Apache Spark Java API, however they cannot take full advantage of
> >> Kotlin language features. With Kotlin Spark API, you can use Kotlin data
> >> classes and lambda expressions.
> >>
> >> The API also adds some helpful extension functions. For example, you can
> >> use `withCached` to perform arbitrary transformations on a Dataset and not
> >> worry about the Dataset unpersisting at the end.
> >>
> >> If you like Kotlin and would like to try the API, we've prepared a Quick
> >> Start Guide to help you set up all the needed dependencies in no time using
> >> either Maven or Gradle:
> >> https://github.com/JetBrains/kotlin-spark-api/blob/master/docs/quick-start-guide.md
> >>
> >> In the repo, you’ll also find a few code examples to get an idea of what
> >> the API looks like:
> >> https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples
> >>
> >> We’d love to see your feedback in the project’s GitHub issues:
> >> https://github.com/JetBrains/kotlin-spark-api/issues.
> >>
> >>
> >> Thanks!
> >>
> >>
> >>
--
Regards,
Pasha
Big Data Tools @ JetBrains
signature.asc
Description: PGP signature
Re: Kotlin Spark API
I just looked at the examples.
https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples
These look v nice! V concise yet flexible. I like the ability to do
inline *side-effects. *E.g. caching or printing or showDs()
package org.jetbrains.spark.api.examples
import org.apache.spark.sql.Row
import org.jetbrains.spark.api.*
fun main() {
withSpark {
val sd = dsOf(1, 2, 3)
sd.createOrReplaceTempView("ds")
spark.sql("select * from ds")
.withCached {
println("asList: ${toList()}")
println("asArray: ${toArray().contentToString()}")
this
}
.to()
.withCached {
println("typed collect: " + (collect() as
Array).contentToString())
println("type collectAsList: " + collectAsList())
}
dsOf(1, 2, 3)
.map { c(it, it + 1, it + 2) }
.to()
.select("_1")
.collectAsList()
.forEach { println(it) }
}
}
So that shows some of the niceness of kotlin: intuitive type conversion
`to`/`to` and `dsOf( list)`- and also the inlining of the side
effects. Overall concise and pleasant to read.
On Tue, 14 Jul 2020 at 12:18, Stephen Boesch wrote:
> I started with scala/spark in 2012 and scala has been my go-to language
> for six years. But I heartily applaud this direction. Kotlin is more like a
> simplified Scala - with the benefits that brings - than a simplified java.
> I particularly like the simplified / streamlined collections classes.
>
> Really looking forward to this development.
>
> On Tue, 14 Jul 2020 at 10:42, Maria Khalusova wrote:
>
>> Hi folks,
>>
>> We would love your feedback on the new Kotlin Spark API that we are
>> working on: https://github.com/JetBrains/kotlin-spark-api.
>>
>> Why Kotlin Spark API? Kotlin developers can already use Kotlin with the
>> existing Apache Spark Java API, however they cannot take full advantage of
>> Kotlin language features. With Kotlin Spark API, you can use Kotlin data
>> classes and lambda expressions.
>>
>> The API also adds some helpful extension functions. For example, you can
>> use `withCached` to perform arbitrary transformations on a Dataset and not
>> worry about the Dataset unpersisting at the end.
>>
>> If you like Kotlin and would like to try the API, we've prepared a Quick
>> Start Guide to help you set up all the needed dependencies in no time using
>> either Maven or Gradle:
>> https://github.com/JetBrains/kotlin-spark-api/blob/master/docs/quick-start-guide.md
>>
>> In the repo, you’ll also find a few code examples to get an idea of what
>> the API looks like:
>> https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples
>>
>> We’d love to see your feedback in the project’s GitHub issues:
>> https://github.com/JetBrains/kotlin-spark-api/issues.
>>
>>
>> Thanks!
>>
>>
>>
Re: Kotlin Spark API
I started with scala/spark in 2012 and scala has been my go-to language for six years. But I heartily applaud this direction. Kotlin is more like a simplified Scala - with the benefits that brings - than a simplified java. I particularly like the simplified / streamlined collections classes. Really looking forward to this development. On Tue, 14 Jul 2020 at 10:42, Maria Khalusova wrote: > Hi folks, > > We would love your feedback on the new Kotlin Spark API that we are > working on: https://github.com/JetBrains/kotlin-spark-api. > > Why Kotlin Spark API? Kotlin developers can already use Kotlin with the > existing Apache Spark Java API, however they cannot take full advantage of > Kotlin language features. With Kotlin Spark API, you can use Kotlin data > classes and lambda expressions. > > The API also adds some helpful extension functions. For example, you can > use `withCached` to perform arbitrary transformations on a Dataset and not > worry about the Dataset unpersisting at the end. > > If you like Kotlin and would like to try the API, we've prepared a Quick > Start Guide to help you set up all the needed dependencies in no time using > either Maven or Gradle: > https://github.com/JetBrains/kotlin-spark-api/blob/master/docs/quick-start-guide.md > > In the repo, you’ll also find a few code examples to get an idea of what > the API looks like: > https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples > > We’d love to see your feedback in the project’s GitHub issues: > https://github.com/JetBrains/kotlin-spark-api/issues. > > > Thanks! > > >
Re: Kotlin Spark API
Is kotlin another new language ? GRADY BOOCH; The United States Department of defence (DOD) is perhaps the largest user of computers in the world. By the mid-1970s, software development for its systems had reached crisis proportions: projects were often late, over budget and they often failed to meet their stated requirements. It was evident that the problems would only worsen as software development costs continued to rise exponentially. To help resolve these problems which were further compounded by the proliferation of hundreds of different languages. The DOD sponsored the development of a single, common high order programming language. The winning design was originally called the Green Language (so called because of its team colour code during the competition), and was renamed ADA On Tue, 14 Jul 2020, 18:42 Maria Khalusova, wrote: > Hi folks, > > We would love your feedback on the new Kotlin Spark API that we are > working on: https://github.com/JetBrains/kotlin-spark-api. > > Why Kotlin Spark API? Kotlin developers can already use Kotlin with the > existing Apache Spark Java API, however they cannot take full advantage of > Kotlin language features. With Kotlin Spark API, you can use Kotlin data > classes and lambda expressions. > > The API also adds some helpful extension functions. For example, you can > use `withCached` to perform arbitrary transformations on a Dataset and not > worry about the Dataset unpersisting at the end. > > If you like Kotlin and would like to try the API, we've prepared a Quick > Start Guide to help you set up all the needed dependencies in no time using > either Maven or Gradle: > https://github.com/JetBrains/kotlin-spark-api/blob/master/docs/quick-start-guide.md > > In the repo, you’ll also find a few code examples to get an idea of what > the API looks like: > https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples > > We’d love to see your feedback in the project’s GitHub issues: > https://github.com/JetBrains/kotlin-spark-api/issues. > > > Thanks! > > >
Kotlin Spark API
Hi folks, We would love your feedback on the new Kotlin Spark API that we are working on: https://github.com/JetBrains/kotlin-spark-api. Why Kotlin Spark API? Kotlin developers can already use Kotlin with the existing Apache Spark Java API, however they cannot take full advantage of Kotlin language features. With Kotlin Spark API, you can use Kotlin data classes and lambda expressions. The API also adds some helpful extension functions. For example, you can use `withCached` to perform arbitrary transformations on a Dataset and not worry about the Dataset unpersisting at the end. If you like Kotlin and would like to try the API, we've prepared a Quick Start Guide to help you set up all the needed dependencies in no time using either Maven or Gradle: https://github.com/JetBrains/kotlin-spark-api/blob/master/docs/quick-start-guide.md In the repo, you’ll also find a few code examples to get an idea of what the API looks like: https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples We’d love to see your feedback in the project’s GitHub issues: https://github.com/JetBrains/kotlin-spark-api/issues. Thanks!
