[
https://issues.apache.org/jira/browse/HUDI-376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xing Pan updated HUDI-376:
--------------------------
Description:
Hi hudi team, it's really encouraging that Hudi is finally officially supported
application on AWS EMR. Great job!
I found a *ClassNotFound* exception when using:
{code:java}
/usr/lib/hudi/bin/run_sync_tool.sh
{code}
in emr master.
And I think is due to demand of aws glue data sdk dependency. (I used aws glue
as hive meta data)
So I added a line to run_sync_tool.sh to get a quick fix for this:
{code:java}
HIVE_JARS=$HIVE_JARS:/usr/lib/hive/auxlib/aws-glue-datacatalog-hive2-client.jar:/usr/share/aws/emr/emr-metrics-collector/lib/aws-java-sdk-glue-1.11.475.jar{code}
not sure if any more jars needed, but these two jar fixed my problem.
I think it would be great if take glue in consideration for emr scripts.
was:
Hi hudi team, it's really encouraging that Hudi is finally officially supported
application on AWS EMR. Great job!
I found a *ClassNotFound* exception when using:
{code:java}
/usr/lib/hudi/bin/run_sync_tool.sh
{code}
in emr master.
And I think is due to demand of aws glue data sdk dependency. (I used aws glue
as hive meta data)
So I added a line to run_sync_tool.sh to get a quick fix for this:
{code:java}
HIVE_JARS=$HIVE_JARS:/usr/lib/hive/auxlib/aws-glue-datacatalog-hive2-client.jar:/usr/share/aws/emr/emr-metrics-collector/lib/aws-java-sdk-glue-1.11.475.jar{code}
not sure if any more jars needed, but these two jar fixed my problem.
> AWS Glue dependency issue for EMR 5.28.0
> ----------------------------------------
>
> Key: HUDI-376
> URL: https://issues.apache.org/jira/browse/HUDI-376
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: CLI
> Reporter: Xing Pan
> Priority: Minor
>
> Hi hudi team, it's really encouraging that Hudi is finally officially
> supported application on AWS EMR. Great job!
> I found a *ClassNotFound* exception when using:
> {code:java}
> /usr/lib/hudi/bin/run_sync_tool.sh
> {code}
> in emr master.
> And I think is due to demand of aws glue data sdk dependency. (I used aws
> glue as hive meta data)
> So I added a line to run_sync_tool.sh to get a quick fix for this:
> {code:java}
> HIVE_JARS=$HIVE_JARS:/usr/lib/hive/auxlib/aws-glue-datacatalog-hive2-client.jar:/usr/share/aws/emr/emr-metrics-collector/lib/aws-java-sdk-glue-1.11.475.jar{code}
> not sure if any more jars needed, but these two jar fixed my problem.
>
> I think it would be great if take glue in consideration for emr scripts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
