[ https://issues.apache.org/jira/browse/SPARK-32521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
fqaiser94 updated SPARK-32521: ------------------------------ Description: The following query currently fails on master brach: {code:scala} sql("SELECT named_struct('a', 1, 'b', 2) a") .select($"a".withField("c", lit(3)).as("a")) .show(false) // java.lang.UnsupportedOperationException: Cannot evaluate expression: with_fields(named_struct(a, 1, b, 2), c, 3) {code} This happens because the Catalyst optimizer tries to statically evaluate the {{WithFields Expression}} (via the {{ConstantFolding}} rule), however it cannot do so because {{WithFields Expression}} is {{Unevaluable}}. The likely solution here is to change {{WithFields Expression}} so that it is not {{foldable}} under any circumstance. This should solve the issue because then Catalyst will no longer attempt to statically evaluate {{WithFields Expression}} anymore. was: The following query currently fails on master brach: {code:scala} sql("SELECT named_struct('a', 1, 'b', 2) a") .select($"a".withField("c", lit(3)).as("a")) .show(false) // java.lang.UnsupportedOperationException: Cannot evaluate expression: with_fields(named_struct(a, 1, b, 2), c, 3) {code} This happens because Catalyst optimizer tries to statically evaluate the {{WithFields Expression}} (via the {{ConstantFolding}} rule), however it cannot do so because {{WithFields Expression}} is {{Unevaluable}}. The likely solution here is to change {{WithFields Expression}} so that it is not {{foldable}} under any circumstance. This should solve the issue because then Catalyst will no longer attempt to statically evaluate {{WithFields Expression}} anymore. > WithFields Expression should not be foldable > -------------------------------------------- > > Key: SPARK-32521 > URL: https://issues.apache.org/jira/browse/SPARK-32521 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.1.0 > Reporter: fqaiser94 > Priority: Major > > The following query currently fails on master brach: > {code:scala} > sql("SELECT named_struct('a', 1, 'b', 2) a") > .select($"a".withField("c", lit(3)).as("a")) > .show(false) > // java.lang.UnsupportedOperationException: Cannot evaluate expression: > with_fields(named_struct(a, 1, b, 2), c, 3) > {code} > This happens because the Catalyst optimizer tries to statically evaluate the > {{WithFields Expression}} (via the {{ConstantFolding}} rule), however it > cannot do so because {{WithFields Expression}} is {{Unevaluable}}. > The likely solution here is to change {{WithFields Expression}} so that it is > not {{foldable}} under any circumstance. This should solve the issue because > then Catalyst will no longer attempt to statically evaluate {{WithFields > Expression}} anymore. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org