[
https://issues.apache.org/jira/browse/SPARK-26664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17571394#comment-17571394
]
Ines Hiverlet commented on SPARK-26664:
---------------------------------------
Hello, are there any news regarding this issue ? Because the problem persists.
The pull request seems to solve the issue but was close waiting for another
idea.
For memory, this was the comment : "I think this is still hacky and it does not
resolve the root issue. Let us think about it and see whether we have a better
solution. I change the target version to 3.0. We will revisit this before the
code freeze of 3.0. If we do not have a solution, we can review this again."
> Make DecimalType's minimum adjusted scale configurable
> ------------------------------------------------------
>
> Key: SPARK-26664
> URL: https://issues.apache.org/jira/browse/SPARK-26664
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Kris Mok
> Priority: Minor
>
> Introduce a new conf flag that allows the user to set the value of
> {{DecimalType.MINIMAL_ADJUSTED_SCALE}}, currently a constant of 6, to match
> their workloads' needs.
> The new flag will be {{spark.sql.decimalOperations.minimumAdjustedScale}}.
> SPARK-22036 introduced a new conf flag
> {{spark.sql.decimalOperations.allowPrecisionLoss}} to match SQL Server's and
> new Hive's behavior of allowing precision loss when performing
> multiplication/division on big and small decimal numbers.
> Along with this feature, a fixed {{MINIMAL_ADJUSTED_SCALE}} was set to 6 for
> when precision loss is allowed.
> Some customer workload may needed a larger adjusted scale to match their
> business needs, and in exchange they may be willing to tolerate some more
> calculations overflowing the max precision, leading to nulls. So they would
> like the minimum adjusted scale to be configurable. Thus the need for a new
> conf.
> The default behavior after introducing this conf flag is not changed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
