[ https://issues.apache.org/jira/browse/SPARK-42857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xinrong Meng updated SPARK-42857: --------------------------------- Description: To reproduce: {code:sh} >>> from decimal import Decimal >>> spark.createDataFrame([Decimal(1.123)]) Traceback (most recent call last): ... pyarrow.lib.ArrowInvalid: Rescaling Decimal128 value would cause data loss >>> {code} The version of PyArrow is 10.0.1; the issue doesn't exist in vanilla PySpark with the same PyArrow version. was: To reproduce: ``` >>> from decimal import Decimal >>> spark.createDataFrame([Decimal(1.123)]) Traceback (most recent call last): ... pyarrow.lib.ArrowInvalid: Rescaling Decimal128 value would cause data loss >>> ``` The version of PyArrow is 10.0.1; the issue doesn't exist in vanilla PySpark with the same PyArrow version. > Unable to CreateDataFrame from Decimal128 > ----------------------------------------- > > Key: SPARK-42857 > URL: https://issues.apache.org/jira/browse/SPARK-42857 > Project: Spark > Issue Type: Story > Components: Connect, PySpark > Affects Versions: 3.4.0 > Reporter: Xinrong Meng > Priority: Major > > To reproduce: > {code:sh} > >>> from decimal import Decimal > >>> spark.createDataFrame([Decimal(1.123)]) > Traceback (most recent call last): > ... > pyarrow.lib.ArrowInvalid: Rescaling Decimal128 value would cause data loss > >>> > {code} > The version of PyArrow is 10.0.1; the issue doesn't exist in vanilla PySpark > with the same PyArrow version. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org