[
https://issues.apache.org/jira/browse/HUDI-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar closed HUDI-260.
-------------------------------
Resolution: Won't Fix
Closing this. Please reopen if needed
> Hudi Spark Bundle does not work when passed in extraClassPath option
> --------------------------------------------------------------------
>
> Key: HUDI-260
> URL: https://issues.apache.org/jira/browse/HUDI-260
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: Spark datasource, SparkSQL Support, Usability
> Reporter: Vinoth Chandar
> Assignee: Udit Mehrotra
> Priority: Major
>
> On EMR's side we have the same findings. *a + b + c +d* work in the following
> cases:
> * The bundle jar (with databricks-avro shaded) is specified using *--jars*
> or *spark.jars* option
> * The bundle jar (with databricks-avro shaded) is placed in the Spark Home
> jars folder i.e. */usr/lib/spark/jars* folder
> However, it does not work if the jar is specified using
> *spark.driver.extraClassPath* and *spark.executor.extraClassPath* options
> which is what EMR uses to configure external dependencies. Although we can
> drop the jar in */usr/lib/spark/jars* folder, but I am not sure if it is
> recommended because that folder is supposed to contain the jars coming from
> spark. Extra dependencies from users side would be better off specified
> through *extraClassPath* option.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
