Re: sql to spark scala rdd

Make sense thanks. Thanks Sri Sent from my iPhone > On 2 Aug 2016, at 03:27, Jacek Laskowski wrote: > > Congrats! > > Whenever I was doing foreach(println) in the past I'm .toDF.show these > days. Give it a shot and you'll experience the feeling yourself! :) > > Pozdrawiam, > Jacek Laskowski

Re: sql to spark scala rdd

Congrats! Whenever I was doing foreach(println) in the past I'm .toDF.show these days. Give it a shot and you'll experience the feeling yourself! :) Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at ht

Re: sql to spark scala rdd

Hi All, Below code calculates cumulative sum (running sum) and moving average using scala RDD type of programming, I was using wrong function which is sliding use scalleft instead. sc.textFile("C:\\Users\\kalit_000\\Desktop\\Hadoop_IMP_DOC\\spark\\data.txt") .map(x => x.split("\\~")) .map(x

Re: sql to spark scala rdd

Hi , I solved it using spark SQL which uses similar window functions mentioned below , for my own knowledge I am trying to solve using Scala RDD which I am unable to. What function in Scala supports window function like SQL unbounded preceding and current row ? Is it sliding ? Thanks Sri Sen

Re: sql to spark scala rdd

hi You mentioned: I already solved it using DF and spark sql ... Are you referring to this code which is a classic analytics: SELECT DATE,balance, SUM(balance) OVER (ORDER BY DATE ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) daily_balance FROM table So how did you solve it using

Re: sql to spark scala rdd

Hi , Just wondering how spark SQL works behind the scenes does it not convert SQL to some Scala RDD ? Or Scala ? How to write below SQL in Scala or Scala RDD SELECT DATE,balance, SUM(balance) OVER (ORDER BY DATE ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT RO

Re: sql to spark scala rdd

Hi, Impossible - see http://www.scala-lang.org/api/current/index.html#scala.collection.Seq@sliding(size:Int,step:Int):Iterator[Repr]. I tried to show you why you ended up with "non-empty iterator" after println. You should really start with http://www.scala-lang.org/documentation/ Pozdrawiam, Ja

Re: sql to spark scala rdd

val test=sc.textFile(file).keyBy(x => x.split("\\~") (0)) .map(x => x._2.split("\\~")) .map(x => ((x(0),x(1),x(2 .map{case (account,datevalue,amount) => ((account,datevalue),(amount.toDouble))}.mapValues(x => x).toArray.sliding(2,1).map(x => (x(0)._1,x(1)._2,(x.foldLeft(0.0)(_ + _._2/x.si

Re: sql to spark scala rdd

Hi All, I already solved it using DF and spark sql I was wondering how to solve in scala rdd, I just got the answer need to check my results compared to spark sql thanks all for your time. I am trying to solve moving average using scala RDD group by key. input:- -987~20150728~100 -987~20150729~

Re: sql to spark scala rdd

Check also this HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: sql to spark scala rdd

Tuple [Lscala.Tuple2;@65e4cb84 On Sun, Jul 31, 2016 at 1:00 AM, Jacek Laskowski wrote: > Hi, > > What's the result type of sliding(2,1)? > > Pozdrawiam, > Jacek Laskowski > > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark > Follow me a

Re: sql to spark scala rdd

Hi, What's the result type of sliding(2,1)? Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Sun, Jul 31, 2016 at 9:23 AM, sri hari kali charan Tummala wrote:

Re: sql to spark scala rdd

tried this no luck, wht is non-empty iterator here ? OP:- (-987,non-empty iterator) (-987,non-empty iterator) (-987,non-empty iterator) (-987,non-empty iterator) (-987,non-empty iterator) sc.textFile(file).keyBy(x => x.split("\\~") (0)) .map(x => x._2.split("\\~")) .map(x => (x(0),x(2)))

Re: sql to spark scala rdd

Hi All, I managed to write using sliding function but can it get key as well in my output ? sc.textFile(file).keyBy(x => x.split("\\~") (0)) .map(x => x._2.split("\\~")) .map(x => (x(2).toDouble)).toArray().sliding(2,1).map(x => (x,x.size)).foreach(println) at the moment my output:-

Re: sql to spark scala rdd

for knowledge just wondering how to write it up in scala or spark RDD. Thanks Sri On Sat, Jul 30, 2016 at 11:24 AM, Jacek Laskowski wrote: > Why? > > Pozdrawiam, > Jacek Laskowski > > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark > Fo

Re: sql to spark scala rdd

Why? Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Sat, Jul 30, 2016 at 4:42 AM, kali.tumm...@gmail.com wrote: > Hi All, > > I managed to write business requ