Re: The equivalent for INSTR in Spark FP

Apologies it should read Jacek. Confused with my friend's name Jared :( Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: The equivalent for INSTR in Spark FP

Thanks Jared for your kind words. I don't think I am anywhere near there yet :) In general I subtract one character before getting to "CD". That is the way debit from debit cards are marked in a Bank's statement. I get out of bound error if -->

Re: The equivalent for INSTR in Spark FP

Congrats! You made it. A serious Spark dev badge unlocked :) Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Tue, Aug 2, 2016 at 9:58 AM, Mich Talebzadeh

Re: The equivalent for INSTR in Spark FP

it should be lit(0) :) rs.select(mySubstr($"transactiondescription", lit(0), instr($"transactiondescription", "CD"))).show(1) +--+ |UDF(transactiondescription,0,instr(transactiondescription,CD))|

Re: The equivalent for INSTR in Spark FP

No thinking on my part!!! rs.select(mySubstr($"transactiondescription", lit(1), instr($"transactiondescription", "CD"))).show(2) +--+ |UDF(transactiondescription,1,instr(transactiondescription,CD))|

Re: The equivalent for INSTR in Spark FP

Thanks Jacek. It sounds like the issue the position of the second variable in substring() This works scala> val wSpec2 = Window.partitionBy(substring($"transactiondescription",1,20)) wSpec2: org.apache.spark.sql.expressions.WindowSpec = org.apache.spark.sql.expressions.WindowSpec@1a4eae2 Using

Re: The equivalent for INSTR in Spark FP

Hi, Interesting... I'm temping to think that substring function should accept the columns that hold the numbers for start and end. I'd love hearing people's thought on this. For now, I'd say you need to define udf to do substring as follows: scala> val mySubstr = udf { (s: String, start: Int,

Re: The equivalent for INSTR in Spark FP

Thanks Jacek, Do I have any other way of writing this with functional programming? select substring(transactiondescription,1,INSTR(transactiondescription,' CD')-2), Cheers, Dr Mich Talebzadeh LinkedIn *

Re: The equivalent for INSTR in Spark FP

Hi Mich, There's no indexOf UDF - http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions$ Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at

Re: The equivalent for INSTR in Spark FP

Any programming expert who can shed some light on this? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

The equivalent for INSTR in Spark FP

Hi, What is the equivalent of FP for the following window/analytic that works OK in Spark SQL This one using INSTR select substring(transactiondescription,1,INSTR(transactiondescription,'CD')-2), select distinct * from ( select