[
https://issues.apache.org/jira/browse/HUDI-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17554873#comment-17554873
]
miomiocat edited comment on HUDI-3696 at 6/17/22 10:38 AM:
-----------------------------------------------------------
[~FredXu]
hi, I'm not sure if your issue is exactly the same as mine, but you can
reference to my own rough modification of this issue to resolve read/write ORC
format with spark
https://github.com/miomiocat/hudi/commit/140b1456151052f8b8ecd576a78b605edcd76c5b
was (Author: cherishc2):
[~FredXu]
hi, I'm not sure if your issue is exactly the same as mine, but you can
reference to my own rough modification of this issue to resolve read/write ORC
format with spark
[https://github.com/miomiocat/hudi/commit/f2ad4f969f56ead8acb82edb7dafb5d408c22f8f]
> ORC dependency conflicts between hudi and spark
> -----------------------------------------------
>
> Key: HUDI-3696
> URL: https://issues.apache.org/jira/browse/HUDI-3696
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: miomiocat
> Priority: Major
>
> Hudi use orc-core-xxx-nohive.jar to initialize orc storage api like
> VectorizedRowBatch but orc-core-xxx.jar in spark, so there are conflicts of
> package dependency between them
>
> {code:java}
> df.write.format("hudi")
> .option("hoodie.table.name", "spark_shell_created_cow_nopartition_type")
> .option("hoodie.datasource.write.precombine.field", "id")
> .option("hoodie.datasource.write.recordkey.field", "id")
> .option("hoodie.datasource.write.partitionpath.field", "id")
> .option("hoodie.datasource.write.hive_style_partitioning", "true")
> .option("hoodie.datasource.write.table.type", "COPY_ON_WRITE")
> .option("hoodie.table.base.file.format", "ORC")
> .mode(Overwrite)
> .save(basePath) {code}
>
> For example, I try to write a ORC format based hudi table with commands
> above, following excetion will be thrown:
>
> {code:java}
> Caused by: java.lang.NoSuchMethodError:
> org.apache.orc.TypeDescription.createRowBatch()Lorg/apache/orc/storage/ql/exec/vector/VectorizedRowBatch;
> at
> org.apache.hudi.io.storage.HoodieOrcWriter.<init>(HoodieOrcWriter.java:84)
> at
> org.apache.hudi.io.storage.HoodieFileWriterFactory.newOrcFileWriter(HoodieFileWriterFactory.java:102)
> at
> org.apache.hudi.io.storage.HoodieFileWriterFactory.getFileWriter(HoodieFileWriterFactory.java:59)
> at
> org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:100)
> at
> org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:73)
> at
> org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:46)
> at
> org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83)
> at
> org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40)
> at
> org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
> at
> org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:134)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more {code}
>
> I tried to replace orc-core-xxx-nohive with orc-core-xxx and fixed some
> import statement, it worked successfully.
> So I think more compatibility modification is needed about orc format and
> spark.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
