[jira] [Commented] (SPARK-40351) Spark Sum increases the precision of DecimalType arguments by 10
[ https://issues.apache.org/jira/browse/SPARK-40351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17630199#comment-17630199 ] Tymofii commented on SPARK-40351: - [~dwsmith1983] make sense. Thank you for pointing out to this doc > Spark Sum increases the precision of DecimalType arguments by 10 > > > Key: SPARK-40351 > URL: https://issues.apache.org/jira/browse/SPARK-40351 > Project: Spark > Issue Type: Question > Components: Optimizer >Affects Versions: 3.2.0 >Reporter: Tymofii >Priority: Minor > > Currently in Spark automatically increases Decimal field by 10 (hard coded > value) after SUM aggregate operation - > [https://github.com/apache/spark/blob/branch-3.2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L1877.] > There are a couple of questions: > # Why was 10 chosen as default one? > # Does it make sense to allow the user to override this value via > configuration? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40351) Spark Sum increases the precision of DecimalType arguments by 10
[ https://issues.apache.org/jira/browse/SPARK-40351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600572#comment-17600572 ] Tymofii commented on SPARK-40351: - # Not sure I understood why you showed those lines of code. # For example, the issue we faced is as follows. Source tables have decimal fields with the certain precision defined during the table creation. There are number of queries, which are used to extract and transform the data from those source tables and load it to the target one, which also has a decimal field with the same precision as in the sources tables. So the users knows for sure, that summing values in the source decimal fields may not result in exceeding the target table field precision. Currently they have to add explicit casting after SUM function to comply with the target table definition since our ETL flow would fail. It may be not very convenient if there are multiple queries. So they could disable automatic increase of the precision in this case for example. # Another question - what is the rationale behind the number 10? > Spark Sum increases the precision of DecimalType arguments by 10 > > > Key: SPARK-40351 > URL: https://issues.apache.org/jira/browse/SPARK-40351 > Project: Spark > Issue Type: Question > Components: Optimizer >Affects Versions: 3.2.0 >Reporter: Tymofii >Priority: Minor > > Currently in Spark automatically increases Decimal field by 10 (hard coded > value) after SUM aggregate operation - > [https://github.com/apache/spark/blob/branch-3.2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L1877.] > There are a couple of questions: > # Why was 10 chosen as default one? > # Does it make sense to allow the user to override this value via > configuration? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-40351) Spark Sum increases the precision of DecimalType arguments by 10
[ https://issues.apache.org/jira/browse/SPARK-40351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tymofii updated SPARK-40351: Description: Currently in Spark automatically increases Decimal field by 10 (hard coded value) after SUM aggregate operation - [https://github.com/apache/spark/blob/branch-3.2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L1877.] There are a couple of questions: # Why was 10 chosen as default one? # Does it make sense to allow the user to override this value via configuration? was: Currently in Spark automatically increases Decimal field by 10 (hard coded value) after SUM aggregate operation - [https://github.com/apache/spark/blob/branch-3.2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L1877.] There are a couple of questions: # Why was 10 chosen as default one? # Is it make sense to allow the user to override this value via configuration? > Spark Sum increases the precision of DecimalType arguments by 10 > > > Key: SPARK-40351 > URL: https://issues.apache.org/jira/browse/SPARK-40351 > Project: Spark > Issue Type: Question > Components: Optimizer >Affects Versions: 3.2.0 >Reporter: Tymofii >Priority: Minor > > Currently in Spark automatically increases Decimal field by 10 (hard coded > value) after SUM aggregate operation - > [https://github.com/apache/spark/blob/branch-3.2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L1877.] > There are a couple of questions: > # Why was 10 chosen as default one? > # Does it make sense to allow the user to override this value via > configuration? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-40351) Spark Sum increases the precision of DecimalType arguments by 10
Tymofii created SPARK-40351: --- Summary: Spark Sum increases the precision of DecimalType arguments by 10 Key: SPARK-40351 URL: https://issues.apache.org/jira/browse/SPARK-40351 Project: Spark Issue Type: Question Components: Optimizer Affects Versions: 3.2.0 Reporter: Tymofii Currently in Spark automatically increases Decimal field by 10 (hard coded value) after SUM aggregate operation - [https://github.com/apache/spark/blob/branch-3.2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L1877.] There are a couple of questions: # Why was 10 chosen as default one? # Is it make sense to allow the user to override this value via configuration? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org