Philip Dodds created SPARK-12128:
------------------------------------
Summary: Multiplication on decimals in dataframe returns null
Key: SPARK-12128
URL: https://issues.apache.org/jira/browse/SPARK-12128
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.5.2, 1.5.1, 1.5.0
Environment: Scala 2.11/Spark 1.5.0/1.5.1/1.5.2
Reporter: Philip Dodds
I hit a weird issue when I tried to multiply to decimals in a select (either in
scala or as SQL), and Im assuming I must be missing the point.
The issue is fairly easy to recreate with something like the following:
{code:java}
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
import org.apache.spark.sql.types.Decimal
case class Trade(quantity: Decimal,price: Decimal)
val data = Seq.fill(100) {
val price = Decimal(20+scala.util.Random.nextInt(10))
val quantity = Decimal(20+scala.util.Random.nextInt(10))
Trade(quantity, price)
}
val trades = sc.parallelize(data).toDF()
trades.registerTempTable("trades")
trades.select(trades("price")*trades("quantity")).show
sqlContext.sql("select
price/quantity,price*quantity,price+quantity,price-quantity from trades").show
{code}
The odd part is if you run it you will see that the addition/division and
subtraction works but the multiplication returns a null.
Tested on 1.5.1/1.5.2 (Scala 2.10 and 2.11)
ie.
{code}
+------------------+
|(price * quantity)|
+------------------+
| null|
| null|
| null|
| null|
| null|
+------------------+
+--------------------+----+--------------------+--------------------+
| _c0| _c1| _c2| _c3|
+--------------------+----+--------------------+--------------------+
|0.952380952380952381|null|41.00000000000000...|-1.00000000000000...|
|1.380952380952380952|null|50.00000000000000...|8.000000000000000000|
|1.272727272727272727|null|50.00000000000000...|6.000000000000000000|
|0.833333333333333333|null|44.00000000000000...|-4.00000000000000...|
|1.000000000000000000|null|58.00000000000000...| 0E-18|
+--------------------+----+--------------------+--------------------+
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
