[
https://issues.apache.org/jira/browse/SPARK-27786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-27786.
---------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 24655
[https://github.com/apache/spark/pull/24655]
> SHA1, MD5, and Base64 expression codegen doesn't work when commons-codec is
> shaded
> ----------------------------------------------------------------------------------
>
> Key: SPARK-27786
> URL: https://issues.apache.org/jira/browse/SPARK-27786
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Minor
> Fix For: 3.0.0
>
>
> When running a custom build of Spark which shades {{commons-codec}}, the
> {{sha1Hex}} expression generates code which doesn't compile:
> {code}
> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator: failed to
> compile: org.codehaus.commons.compiler.CompileException: File
> 'generated.java', Line 47, Column 93: A method named "sha1Hex" is not
> declared in any enclosing class nor any supertype, nor through a static import
> {code}
> This is caused by an interaction between Spark's code generator and the
> shading: the current code generator embeds
> "org.apache.commons.codec.digest.DigestUtils.sha1Hex" into a larger codegen
> template, preventing JarJarLinks from being able to replace it with the
> shaded class's name. The generated code ends up using the unshaded name but
> the unshaded dependency isn't on our classpath, triggering the above
> compilation error.
> To fix this problem and allow for proper shading, we can replace the
> hardcoded string literal with {{classof[DigestUtils].getName}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
