codope commented on PR #5674:
URL: https://github.com/apache/hudi/pull/5674#issuecomment-1143412485
> LGTM. Better have some e2e test on this bundle jar alone to validate the
functionality.
Running this bundle jar with run_sync_tool was throwing below error
```
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.parquet.avro.AvroSchemaConverter.convert(Lorg/apache/parquet/schema/MessageType;)Lorg/apache/avro/Schema;
at
org.apache.hudi.common.table.TableSchemaResolver.convertParquetSchemaToAvro(TableSchemaResolver.java:351)
at
org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaFromDataFile(TableSchemaResolver.java:158)
at
org.apache.hudi.common.table.TableSchemaResolver.hasOperationField(TableSchemaResolver.java:575)
at
org.apache.hudi.common.table.TableSchemaResolver.<init>(TableSchemaResolver.java:83)
at
org.apache.hudi.sync.common.AbstractSyncHoodieClient.getDataSchema(AbstractSyncHoodieClient.java:164)
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:196)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:142)
at
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:130)
at
org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool.main(AwsGlueCatalogSyncTool.java:68)
```
Please check the last commit where I explicitly added parquet-avro
dependencies and shaded from all other modules. After that I was able to run
run_sync_tool with this bundle.
<img width="1172" alt="Screenshot 2022-06-01 at 3 48 21 PM"
src="https://user-images.githubusercontent.com/16440354/171382549-f876b8b6-75d3-41a0-bae3-29269869c902.png";>
Note: These dependencies are not really required if we run it together with
hudi-utilities-slim-bundle and hudi-spark-bundle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
![https://user-images.githubusercontent.com/16440354/171382549-f876b8b6-75d3-41a0-bae3-29269869c902.png"](https://user-images.githubusercontent.com/16440354/171382549-f876b8b6-75d3-41a0-bae3-29269869c902.png"){kind=link}