Creating a dataframe with decimals changes the precision and scale

[Philip Dodds]

I'm not sure if there is a way around this just looking for advice,

I create a dataframe from some decimals with a specific precision and
scale,  then when I look at the dataframe it has defaulted the precision
and scale back again.

Is there a way to retain the precision and scale when doing a toDF()

example code:

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
import org.apache.spark.sql.types.Decimal

val a = new Decimal().set(BigDecimal(50),14,4)
val b = new Decimal().set(BigDecimal(50),14,4)

val data = Seq.fill(5) {
     (a,b)
   }

val trades = data.toDF()

trades.printSchema()

the result of this code would show

root
 |-- _1: decimal(38,18) (nullable = true)
 |-- _2: decimal(38,18) (nullable = true)

sqlContext: org.apache.spark.sql.SQLContext =
org.apache.spark.sql.SQLContext@3a1a48f7
import sqlContext.implicits._
import org.apache.spark.sql.types.Decimal
a: org.apache.spark.sql.types.Decimal = 50.000000000000000000
b: org.apache.spark.sql.types.Decimal = 50.000000000000000000
data: Seq[(org.apache.spark.sql.types.Decimal,
org.apache.spark.sql.types.Decimal)] =
List((50.000000000000000000,50.000000000000000000),
(50.000000000000000000,50.000000000000000000),
(50.000000000000000000,50.000000000000000000),
(50.000000000000000000,50.000000000000000000),
(50.000000000000000000,50.000000000000000000))
trades: org.apache.spark.sql.DataFrame = [_1: decimal(38,18), _2:
decimal(38,18)]


Any advice would be brilliant


Thanks

P


-- 
Philip Dodds

philip.do...@gmail.com
@philipdodds