yihua opened a new pull request, #9963:
URL: https://github.com/apache/hudi/pull/9963
### Change Logs
This PR fixes the bundling of RoaringBitmap dependency in Hudi bundles by
including it in the shade rules and shading the classes, to avoid dependency
conflict with engine-provided jars, e.g., Spark.
Before this fix, with Hudi Spark bundle, the following exception is thrown
by Spark 3.2:
```
Caused by: java.lang.NoSuchMethodError:
org.roaringbitmap.longlong.Roaring64NavigableMap.serializePortable(Ljava/io/DataOutput;)V
at
org.apache.hudi.common.table.log.LogReaderUtils.encodePositions(LogReaderUtils.java:128)
at
org.apache.hudi.common.table.log.LogReaderUtils.encodePositions(LogReaderUtils.java:109)
at
org.apache.hudi.common.table.log.block.HoodieLogBlock.addRecordPositionsToHeader(HoodieLogBlock.java:147)
at
org.apache.hudi.common.table.log.block.HoodieDeleteBlock.<init>(HoodieDeleteBlock.java:88)
at
org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:480)
at
org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:454)
at
org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:83)
at
org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:313)
... 28 more
```
### Impact
Fix the bundling of RoaringBitmap dependency.
The bundle size is only slightly increased (~400KB):
Before
```
24069469 Oct 31 14:45 hudi-spark3.3-bundle_2.12-1.0.0-SNAPSHOT-sources.jar
104788296 Oct 31 14:45 hudi-spark3.3-bundle_2.12-1.0.0-SNAPSHOT.jar
10535 Oct 31 14:44
original-hudi-spark3.3-bundle_2.12-1.0.0-SNAPSHOT-sources.jar
14600 Oct 31 14:44
original-hudi-spark3.3-bundle_2.12-1.0.0-SNAPSHOT.jar
```
After
```
24345617 Oct 31 15:13 hudi-spark3.3-bundle_2.12-1.0.0-SNAPSHOT-sources.jar
105241434 Oct 31 15:13 hudi-spark3.3-bundle_2.12-1.0.0-SNAPSHOT.jar
10535 Oct 31 15:12
original-hudi-spark3.3-bundle_2.12-1.0.0-SNAPSHOT-sources.jar
14627 Oct 31 15:12
original-hudi-spark3.3-bundle_2.12-1.0.0-SNAPSHOT.jar
```
### Risk level
none
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
