Not quiet sure whats happening, but its not an issue with multiplication i guess as the following query worked for me:
trades.select(trades("price")*9.5).show +-------------+ |(price * 9.5)| +-------------+ | 199.5| | 228.0| | 190.0| | 199.5| | 190.0| | 256.5| | 218.5| | 275.5| | 218.5| ...... ...... Could it be with the precision? ccing dev list, may be you can open up a jira for this as it seems to be a bug. Thanks Best Regards On Mon, Nov 30, 2015 at 12:41 AM, Philip Dodds <philip.do...@gmail.com> wrote: > I hit a weird issue when I tried to multiply to decimals in a select > (either in scala or as SQL), and Im assuming I must be missing the point. > > The issue is fairly easy to recreate with something like the following: > > > val sqlContext = new org.apache.spark.sql.SQLContext(sc) > import sqlContext.implicits._ > import org.apache.spark.sql.types.Decimal > > case class Trade(quantity: Decimal,price: Decimal) > > val data = Seq.fill(100) { > val price = Decimal(20+scala.util.Random.nextInt(10)) > val quantity = Decimal(20+scala.util.Random.nextInt(10)) > > Trade(quantity, price) > } > > val trades = sc.parallelize(data).toDF() > trades.registerTempTable("trades") > > trades.select(trades("price")*trades("quantity")).show > > sqlContext.sql("select > price/quantity,price*quantity,price+quantity,price-quantity from > trades").show > > The odd part is if you run it you will see that the addition/division and > subtraction works but the multiplication returns a null. > > Tested on 1.5.1/1.5.2 (Scala 2.10 and 2.11) > > ie. > > +------------------+ > > |(price * quantity)| > > +------------------+ > > | null| > > | null| > > | null| > > | null| > > | null| > > +------------------+ > > > +--------------------+----+--------------------+--------------------+ > > | _c0| _c1| _c2| _c3| > > +--------------------+----+--------------------+--------------------+ > > |0.952380952380952381|null|41.00000000000000...|-1.00000000000000...| > > |1.380952380952380952|null|50.00000000000000...|8.000000000000000000| > > |1.272727272727272727|null|50.00000000000000...|6.000000000000000000| > > |0.833333333333333333|null|44.00000000000000...|-4.00000000000000...| > > |1.000000000000000000|null|58.00000000000000...| 0E-18| > > +--------------------+----+--------------------+--------------------+ > > > Just keen to know what I did wrong? > > > Cheers > > P > > -- > Philip Dodds > > >