[
https://issues.apache.org/jira/browse/HUDI-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931817#comment-16931817
]
Udit Mehrotra commented on HUDI-254:
------------------------------------
[~vinoth] On EMR's side we have the same findings. *a + b + c +d* work in the
following cases:
* The bundle jar (with databricks-avro shaded) is specified using *--jars* or
*spark.jars* option
* The bundle jar (with databricks-avro shaded) is placed in the Spark Home
jars folder i.e. */usr/lib/spark/jars* folder
However, it does not work if the jar is specified using
*spark.driver.extraClassPath* and *spark.executor.extraClassPath* options which
is what EMR uses to configure external dependencies. Although we can drop the
jar in */usr/lib/spark/jars* folder, but I am not sure if it is recommended
because that folder is supposed to contain the jars coming from spark. Extra
dependencies from users side would be better off specified through
*extraClassPath* option.
> Provide mechanism for installing hudi-spark-bundle onto an existing spark
> installation
> --------------------------------------------------------------------------------------
>
> Key: HUDI-254
> URL: https://issues.apache.org/jira/browse/HUDI-254
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: Spark datasource, SparkSQL Support
> Reporter: Vinoth Chandar
> Assignee: Vinoth Chandar
> Priority: Major
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> A lot of discussions around this kicked off from
> [https://github.com/apache/incubator-hudi/issues/869]
> Breaking down into phases, when we drop the hudi-spark-bundle*.jar onto the
> `jars` folder
>
> a) Writing data via Hudi datasource should work
> b) Spark datasource reads should work
>
> c) a + Hive Sync should work
> d) SparkSQL on Hive synced table works
>
> Start with Spark 2.3 (current demo setup) and then proceed to 2.4 and iron
> out issues.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
