[
https://issues.apache.org/jira/browse/SPARK-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Dodds closed SPARK-12128.
--------------------------------
Resolution: Invalid
The issue is the promotion of the scale, since the default BigDecimal is a
precision of 38, and a scale of 19 the multiplication will promote that using
the logic defined.
This will mean that (since 38 is the max precision) the scale will be the two
scales added together (36) meaning the result is (38,36) and thus we get an
overflow.
I assume the solution is to cast the two decimals going in to ensure the result
has enough, ie.
SELECT cast(a as decimal(38,9))*cast(b as decimal(38,9)) FROM t
> Multiplication on decimals in dataframe returns null
> ----------------------------------------------------
>
> Key: SPARK-12128
> URL: https://issues.apache.org/jira/browse/SPARK-12128
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.5.0, 1.5.1, 1.5.2
> Environment: Scala 2.11/Spark 1.5.0/1.5.1/1.5.2
> Reporter: Philip Dodds
>
> I hit a weird issue when I tried to multiply to decimals in a select (either
> in scala or as SQL), and Im assuming I must be missing the point.
> The issue is fairly easy to recreate with something like the following:
> {code:java}
> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
> import sqlContext.implicits._
> import org.apache.spark.sql.types.Decimal
> case class Trade(quantity: Decimal,price: Decimal)
> val data = Seq.fill(100) {
> val price = Decimal(20+scala.util.Random.nextInt(10))
> val quantity = Decimal(20+scala.util.Random.nextInt(10))
> Trade(quantity, price)
> }
> val trades = sc.parallelize(data).toDF()
> trades.registerTempTable("trades")
> trades.select(trades("price")*trades("quantity")).show
> sqlContext.sql("select
> price/quantity,price*quantity,price+quantity,price-quantity from trades").show
> {code}
> The odd part is if you run it you will see that the addition/division and
> subtraction works but the multiplication returns a null.
> Tested on 1.5.1/1.5.2 (Scala 2.10 and 2.11)
> ie.
> {code}
> +------------------+
> |(price * quantity)|
> +------------------+
> | null|
> | null|
> | null|
> | null|
> | null|
> +------------------+
> +--------------------+----+--------------------+--------------------+
> | _c0| _c1| _c2| _c3|
> +--------------------+----+--------------------+--------------------+
> |0.952380952380952381|null|41.00000000000000...|-1.00000000000000...|
> |1.380952380952380952|null|50.00000000000000...|8.000000000000000000|
> |1.272727272727272727|null|50.00000000000000...|6.000000000000000000|
> |0.833333333333333333|null|44.00000000000000...|-4.00000000000000...|
> |1.000000000000000000|null|58.00000000000000...| 0E-18|
> +--------------------+----+--------------------+--------------------+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
