[
https://issues.apache.org/jira/browse/SPARK-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takeshi Yamamuro closed SPARK-20390.
------------------------------------
Resolution: Not A Problem
> Non-deterministic expressions could exist in grouping keys
> ----------------------------------------------------------
>
> Key: SPARK-20390
> URL: https://issues.apache.org/jira/browse/SPARK-20390
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Takeshi Yamamuro
>
> Deterministic expressions only exist in grouping keys though,
> non-deterministic one could exist there in some cases.
> This is because `AttributeReference` does not respect `deterministic`
> properties in query plans.
> A example is as follows;
> {code}
> scala> val df = sql("""select rand(0), count(1) group by 1""")
> df: org.apache.spark.sql.DataFrame = [rand(0): double, count(1): bigint]
> scala> df.explain(true)
> == Parsed Logical Plan ==
> 'Aggregate [1], [unresolvedalias('rand(0), None), unresolvedalias('count(1),
> None)]
> +- OneRowRelation$
> == Analyzed Logical Plan ==
> rand(0): double, count(1): bigint
> Aggregate [_nondeterministic#92], [_nondeterministic#92 AS rand(0)#90,
> count(1) AS count(1)#91L]
> +- Project [rand(0) AS _nondeterministic#92]
> +- OneRowRelation$
> == Optimized Logical Plan ==
> Aggregate [_nondeterministic#92], [_nondeterministic#92 AS rand(0)#90,
> count(1) AS count(1)#91L]
> +- Project [rand(0) AS _nondeterministic#92]
> +- OneRowRelation$
> == Physical Plan ==
> *HashAggregate(keys=[_nondeterministic#92], functions=[count(1)],
> output=[rand(0)#90, count(1)#91L])
> +- Exchange hashpartitioning(_nondeterministic#92, 200)
> +- *HashAggregate(keys=[_nondeterministic#92],
> functions=[partial_count(1)], output=[_nondeterministic#92, count#94L])
> +- *Project [rand(0) AS _nondeterministic#92]
> +- Scan OneRowRelation[]
> scala> df.show
> +------------------+--------+
> | rand(0)|count(1)|
> +------------------+--------+
> |0.8446490682263027| 1|
> +------------------+--------+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
