[ https://issues.apache.org/jira/browse/SPARK-17230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-17230: ------------------------------------ Assignee: Apache Spark (was: Davies Liu) > Writing decimal to csv will result empty string if the decimal exceeds (20, > 18) > ------------------------------------------------------------------------------- > > Key: SPARK-17230 > URL: https://issues.apache.org/jira/browse/SPARK-17230 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.6.2, 2.0.0 > Reporter: Davies Liu > Assignee: Apache Spark > > {code} > // file content > spark.read.csv("/mnt/djiang/test-case.csv").show > // read in as string and create temp view > spark.read.csv("/mnt/djiang/test-case.csv").createOrReplaceTempView("test") > // confirm schema > spark.table("test").printSchema > // apply decimal calculation, confirm the result is correct > spark.sql("select _c0, cast(_c0 as long) * cast('1.0' as decimal(38, 18)) > from test").show(false) > // run the same query, and write out as csv > spark.sql("select _c0, cast(_c0 as long) * cast('1.0' as decimal(38, 18)) > from test").write.csv("/mnt/djiang/test-case-result") > // show the content of the result file, particularly, for number exceeded > decimal(20, 18), the csv is not writing anything or failing silently > spark.read.csv("/mnt/djiang/test-case-result").show > +------+ > | _c0| > +------+ > | 1| > | 10| > | 100| > | 1000| > | 10000| > |100000| > +------+ > root > |-- _c0: string (nullable = true) > +------+-------------------------------------------------------------------------------------------------------------------------------------------------+ > > |_c0 |(CAST(CAST(CAST(CAST(_c0 AS DECIMAL(20,0)) AS BIGINT) AS DECIMAL(20,0)) > AS DECIMAL(38,18)) * CAST(CAST(1.0 AS DECIMAL(38,18)) AS DECIMAL(38,18)))| > +------+-------------------------------------------------------------------------------------------------------------------------------------------------+ > > |1 |1.000000000000000000 | > |10 |10.000000000000000000 | > |100 |100.000000000000000000 | > |1000 |1000.000000000000000000 | > |10000 |10000.000000000000000000 | > |100000|100000.000000000000000000 | > +------+-------------------------------------------------------------------------------------------------------------------------------------------------+ > +------+--------------------+ > | _c0| _c1| > +------+--------------------+ > | 1|1.000000000000000000| > | 10|10.00000000000000...| > | 100| | > | 1000| | > | 10000| | > |100000| | > +------+--------------------+ > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org